# Agent Squad Substrate: the full journey

> Markdown version of **https://playgrounds.digitalhubassist.ai/substrate-journey-motor-v3-en.html**
>
> Generated automatically from the HTML. The web page is the canonical source: the 16 SVG diagrams (pipeline, DAGs, architecture) have no equivalent in plain Markdown and are marked in the text.

---

*Convergence Hub · agents-platform · 2026-05-19*

## A *symbolic* substrate for agents that *compose*

How, in a single session, we went from one speaker's insight, “control planes, not Photoshops”, to four productive workflows running with a real LLM on a graph of typed claims in Hetzner.

| | |
|---|---|
| **10** | Primitives |
| **16** | Operations |
| **4** | Templates |
| **$0.06** | Smoke total |
| **28/28** | Tests green |

*Figures from the session of May 19, 2026. The current state is in chapter 15.*

> **Current state · 2026-10-26.** This document is a chronicle: each chapter is a dated snapshot, starting with the session of May 19, 2026 that this cover describes. Today's system is in chapter 15.
>
> The "v3" in the address is the third edition of this document, not the engine version. The engine running in production is v2, the only one since September 5, 2026.
>
> Why Inngest and not a cron job in Postgres? The answer, with evidence from the code and from production.

*Before the journey · The obvious question*

## What is a *substrate*?

If you are coming to this for the first time, that word is probably the first thing that stops you. Spend five minutes here and the rest of the document reads differently.

A **substrate** is the layer beneath the agents and workflows where *everything they produce ends up typed and queryable*. It is not a database. It is not a knowledge base with embeddings. It is a graph where every decision, every output, every human judgment, and every inference an agent asserts is deposited as a node with explicit relationships to the others.

The difference from a conversational assistant is direct: the assistant is *amnesiac*, the substrate is *consistent with its own past*. Every new answer can consult everything that was decided before and reason against it. The approval you gave yesterday is context available to the agent that starts tomorrow.

**Without a substrate**: Every workflow is amnesiac. What it produces is lost.

```
**Workflow A** → output to the user
 ↓ (nothing is recorded)
 **Workflow B** starts with no context
 → repeats questions, contradicts earlier decisions
```

- Every conversation starts from zero, even between agents on the same team
- The user acts as the system's memory (you paste context into every prompt)
- Decisions evaporate: nobody knows who approved what, or why
- Claims are impossible to audit: the model said X, and nobody knows where it got it

**With a substrate**: Every workflow deposits traceable inference. What it produces compounds value.

```
**Workflow A** → outputs + typed Claims
 ↓ (the graph grows)
 **Workflow B** reads A's Claims before starting
 → reasons against earlier decisions, not against zero
```

- Every human decision materializes as a Claim that future workflows can query
- The agent team accumulates structured memory: not just logs, typed knowledge
- Native traceability: any claim has provenance (which Trace produced it, which human validated it)
- Network effects: every extra workflow adds context to all the ones that follow

> *[SVG diagram: see the web version]*

*Chapter one · The origin*

## It started with a *talk* at Stanford

You arrived at the session with the transcript of Chamath Palihapitiya at the Stanford AI Club. One specific sentence, *"I would rather build MS-DOS and Windows, not Adobe Photoshop"*, was the lever that turned the problem around.

> *[SVG diagram: see the web version]*

Tools wear out; substrates compound. What Chamath called a *control plane* was not a product but a layer beneath the agents where every decision, claim, intent, and artifact is traced and available for future inferences. That is what makes a team of agents get better with use.

The point was simple and hard at once: if I build the consumer Agent Squad without a substrate, I repeat the pattern of every AI SaaS: users produce pixels but the system does not learn. If I build the substrate *first*, every workflow I mount on top of it afterward compounds value in the graph. The consumer Agent Squad becomes *the first ontology overlay*, not the final product.

And then came the strategic piece. Chamath was connecting control plane with network effect: the graph of typed Claims is what creates the cross company compounding effect, not the app mounted on top.

We decided not to build a Photoshop. The product is the substrate. The app Miles is finishing is the first public demonstration of what that substrate makes possible. The underlying idea: `{ Intent, Plan, Trace, Claim, Artifact }` are the primitives that any agentic workflow can compile into an executable DAG, and any human decision becomes typed context for the next one.

*Chapter two · The seven forks*

## Seven *strategic* decisions made at speed

Before touching a single line of code, we aligned on seven forks. Each one closes a branch of the tree that opens another. All of them were decided in the same session.

*Chapter three · The vocabulary*

## Ten *primitives* and two hard rules

The entire substrate is expressed in ten types. There is nothing else. If something does not fit here, it is not part of the substrate: it is an ontology overlay built on top.

- **Intent**: What someone (human or agent) wants to happen. It has kind, subject, constraints, acceptance_criteria_ref.
- **Plan**: A DAG of Steps compiled from an Intent. Immutable once compiled; a new version means a new Plan.
- **Step**: A node of the Plan that references an Operation, with specific inputs and an expected output schema.
- **Operation**: A Capability from the catalog (e.g. text.compose_brief@1.0.0). Declares knowledge_access for Rule B.
- **Trace**: A concrete execution of a Plan. Every executed Step generates a StepExecution with inputs/outputs/cost.
- **Claim**: A semantic triple (subject, predicate, object) with provenance and confidence. Example: approvedBy → human:roberto.
- **Artifact**: The materialized output of a Trace. Content addressed, with a status (pending_review, approved, rejected) and an embedding.
- **Evaluator**: A pure function over a Step's output. Emits a verdict {pass|fail, score, rationale}. Lives in the spec.
- **Ontology Overlay**: The vocabulary and usage patterns of a domain (e.g. the Agent Squad consumer app). Built on top of the substrate.
- **Workspace Manifest**: A cached snapshot of the workspace state that agents read on startup. Materialized nightly.

> **Rule A · Inviolable: WorkspaceManifest is a *primitive*, not a convention**
>
> Every agent reads the cached manifest on startup, not the full graph. If the manifest does not exist, the agent refuses to execute. This forces nightly precomputation and eliminates unstable graph queries on the hot path.

> **Rule B · Inviolable: Manifest first contract: `knowledge_access.requires_vector`**
>
> Every Operation declares whether it needs vector search. The runtime rejects execution if an Op attempts a vector lookup without having declared it. It cuts pgvector queries by about 70% and makes it auditable which Operations are expensive to read.

*Chapter four · The tech stack*

## Twelve *tech* decisions made in an hour

Every layer has a concrete reason. No technology was chosen out of inertia or hype. The default question was: what mature, cheap thing covers this without painting us into a corner?

| Layer | Tech | Why |
|---|---|---|
| Spec | JSON Schema + Zod + TypeScript | Open spec, not a proprietary DSL. Anyone with valid JSON Schema outputs without touching the engine. |
| Engine | v2 event sourced + Inngest OSS | The compiler produces graph_nodes; a pure reducer + apply event project runs/run_events; a worker consumes work_items. Inngest keeps durable events and crons. execute-plan is the v1 executor being deprecated. |
| Substrate DB | Postgres 16 + pgvector + pgvectorscale | DiskANN for 3072 dim embeddings (HNSW tops out at 2000). Dedicated on Hetzner, NOT on InsForge. |
| Workspace DB | InsForge | Still used for the consumer app (auth, user data, outputs feed). Separate from the substrate by design. |
| LLM | Max CLI (claude-cli, Max plan; no Anthropic API) | Sonnet 4.5 and Opus for composers and scorers; Grok for synthesis. The Vercel AI SDK + @ai-sdk/anthropic path stayed as a dormant gateway (Phase 1). |
| Embedding | OpenAI text-embedding-3-large (3072d) | Best precision/recall for semantic retrieval. Backup to R2 from day one in case we migrate vector DBs. |
| Reranker | Cohere Rerank v3.5 | Relevance layer over the top K. Can be switched on when the graph grows. Not needed yet. |
| Evaluators | In house in packages/substrate-spec | No LangSmith. Evaluators are tested code that lives with the spec, not an external service. |
| Observability | Langfuse OSS v3 self hosted | Web + worker + ClickHouse + MinIO + Redis + Postgres. Tracing by the substrate's trace_id. |
| API runtime | Hono on Bun 1.3 | Zero cold start, minimal footprint. A single Hono app exposes /api/intents, /api/approvals, /api/health. |
| API contract | Internal tRPC · external REST + OpenAPI 3.1 | Shared types internally. OpenAPI externally so any client can implement against it. |
| Compute | Apps on Vercel · Engine + DB + Obs on Hetzner | Hetzner 4CPU/8GB already paid for. The stateful engine lives where we have disk; the stateless apps stay on Vercel. |

*Chapter five · Infrastructure stood up*

## A single Hetzner box, every service *switched on*

It is not the cloud, it is not Kubernetes, it is not serverless. It is a single machine with systemd and three Docker Compose stacks living together peacefully. Total memory used: roughly 1.7 GB out of 8 available.

The backup runs nightly at 3 AM via cron over `~/substrate-infra/scripts/backup-substrate-db.sh`. It generates a dump into `~/backups/substrate/` and, if the rclone remote `r2-substrate:` exists, also pushes it to Cloudflare R2. The substrate's SQL migrations live in `agent-squad-app/db/substrate/migrations/0001_init.sql`: 15 tables, DiskANN indexes over the vectors of Claims and Artifacts, and date partitioning on `traces` and `step_executions`.

Think of it this way: imagine the brief you approved yesterday hanging on a wall. Next to it, *post-its* keep appearing that the system writes automatically. Each of those post-its is **a Claim**.

| Piece | What it answers | Examples |
|---|---|---|
| subject | What thing the post-it is stuck to | a brief · your voice · a project · a decision · a prospect |
| predicate | What it says about that thing | `approvedBy` · `hasKind` · `voiceTone` · `fitScore` · `producedByOp` |
| object | What value it carries | `"human:roberto"` · `"doc"` · `"direct, no detours"` · `0.75` |
| provenance | Where it came from (native audit) | which Trace emitted it · which Step · which human confirmed it |

> *[SVG diagram: see the web version]*

## Four *known* alternatives and where they break for this case

### Vanilla RAG

*retrieval-augmented generation*

**What it does well.** Retrieving chunks of documents that *resemble* your query using vector similarity. Good for Q&A over a static corpus (manuals, knowledge bases).

**Where it breaks here.** It cannot answer *"what did Roberto decide about this topic?"*; it only finds documents that *mention* the topic. Zero auditability: the LLM cites a chunk but you do not know whether it was a decision, a rejected opinion, or an old hypothesis. Without structure, every answer can hallucinate about claims that were never made.

> RAG tells you **what looks similar**. The substrate tells you **what was asserted, by whom, and when**.

### LangChain · LlamaIndex

*orchestration libraries*

**What it does well.** Composing chains of prompts and LLM calls with a consistent API. Useful for quickly prototyping linear pipelines in a notebook.

**Where it breaks here.** State lives in process memory. If the server crashes midway through the chain, you start from zero. It has *no* durable human in the loop, *no* per step retries with persisted backoff, *no* graph of claims; it is just glue between prompts. For production flows longer than 2 minutes it becomes fragile.

> LangChain is **the glue between prompts**. The substrate is **the durable engine + the structured memory**.

### AutoGPT · BabyAGI · CrewAI

*autonomous agent frameworks*

**What it does well.** Viral demos: agents that prompt themselves in a loop, pick tools, break down tasks. Spectacular on YouTube.

**Where it breaks here.** The LLM decides the next step on every iteration, so cost and latency are unpredictable, and runs go on forever when it gets confused. *Without* an explicit DAG you cannot audit what it will do or bound the blast radius. *Without* shared memory between runs, every conversation starts from zero. It works in demos, not in operation.

> Agent frameworks are **improvisation**. The substrate is **a score with room for improvisation inside each Step**.

### Workflow engine alone

*Temporal · Inngest · Step Functions*

**What it does well.** Durable execution, retries, human in the loop, observability. It is the foundation of what we are using; Inngest itself falls in this category.

**Where it breaks here.** It is *orchestration only*. It has no semantic notion of what is being asserted. Each Step runs loose logic; the output ends up as a blob in a generic table. Without the typed Claims layer on top, two workflows in the same domain cannot share claims in a structured way, only events.

> The workflow engine is **the verb**. The substrate adds **the noun (typed Claims)** on top. Inngest + Postgres + spec = substrate.

*Chapter six · The catalog*

## A library of *capabilities* that any Plan draws from

The catalog is public (packages/substrate-spec). Every new PlanTemplate adds between zero and four domain specific Operations; the rest of the engine (durable execution, human gate, lineage, claims, cost tracking) comes for free.

- **73 Operations**: Atomic capabilities registered in the engine. Each one declares inputs, outputs, knowledge_access, and a cost model. The largest domain is **research (17)**; the latest to join is research.audience_intelligence.
- **12 Evaluators**: Pure functions that judge outputs. They emit a verdict {pass|fail, score, rationale}.
- **23 PlanTemplates**: Reusable DAGs that compile an Intent into a Plan. The 4 canonical ones (below) plus document-extract/query, junta-yt-review, scene-production, and the video pipeline (script/assembly/publish/reel).
- **✓ Blocking gate**: Without green, nothing is merged or deployed: check + unit (web/api) + e2e-criticos + e2e-fast/3d in the sovereign CI. Every Operation and PlanTemplate is gated there.

- research.synthesize@1.0.0
- research.keyword_expand@1.0.0
- research.reddit_search@1.0.0
- research.youtube_comments@1.0.0
- research.rank_opportunities@1.0.0
- research.audience_intelligence@1.0.0

- document.ingest@1.0.0
- document.chunk@1.0.0
- document.extract@1.0.0
- document.query@1.0.0

- media.ingest@1.0.0
- media.transcribe@1.0.0
- url.fetch_transcript@1.0.0

- video.script_draft@1.0.0
- broll.generate@1.0.0
- heygen.generate@1.0.0
- voice.tts@1.0.0

- junta.panel@1.0.0
- junta.compose_verdict@1.0.0
- junta.load_report@1.0.0

- text.compose_narrative@2.0.0
- text.compose_brief@1.0.0

- trace.query@1.0.0
- artifact.publish@1.0.0
- claim.recall_decisions@1.0.0
- claim.recall_voice@1.0.0

- evaluator.run@1.0.0

- prospect.search@1.0.0
- pricing.observe@1.0.0
- metrics.digest@1.0.0
- audience.load@1.0.0

A representative sample. The full catalog is **73 Operations** registered in the engine (inngest/operations), with research as the densest domain.

*Chapter seven · The four Plan Templates*

## Four productive *DAGs*, each with its own agent

Each PlanTemplate is a topologically sorted DAG compiled from an Intent. The colors mark the Step type: amber LLM, cyan evaluator, green publish, pink human gate, gray system.

> *[SVG diagram: see the web version]*

*Chapter eight · Human gate suspension*

## The *durable* gate where the person has the final say

In v2 the gate is not a `step.waitForEvent`: `ACTIVAR_GATE` creates a durable flow in `artifact_approval_flows`. Review, approval/choice, executor acknowledgment, and closure are persisted states; `deadline_at` plus the sweep apply the fallback with CAS, and `required_role` defines who can decide.

**Critical bug we learned from:** the match in `step.waitForEvent` must use `async.data.X`, not `event.data.X`. `event` references the original trigger (`plan.compiled`); using it there evaluates at the *time of the suspension*, not of the future event, and matches against null. `async.data` references the event being waited on. Without this, every gate hangs indefinitely.

**What this unlocks:** the human can approve/reject/add a comment, and the decision becomes a typed Claim that future workflows can query. The next time we assemble a brief on the same topic, `claim.recall_decisions` already has "this angle won" as context. This is the compounding effect of the substrate: it is not retrieval, it is the construction of structured memory.

*Chapter nine · Smoke test with a real LLM*

## Four templates *activated* with real Sonnet 4.5

We wired ANTHROPIC_API_KEY + OPENAI_API_KEY into apps/api/.env, restarted systemd, and fired one Intent per template. All four reached awaiting_human with real outputs. Total cost for the four runs: six cents.

**The non trivial part:** Sonnet reasoned about the workspace's own backlog. The standup-digest narrative literally mentioned how many earlier digests were left `queued` and `awaiting_human`, because `trace.query` + `artifact.list_recent` + `claim.recall` fed it that data. The retrieval → composer chain is validated end to end with real substrate data.

*Chapter ten · The closed loop*

## We approved the *brief* and five Claims were born

The last step of the session was to close the content-brief loop: POST /api/approvals with decision approve and a comment. The gate reactivated, the artifact moved to approved, and the substrate materialized five typed Claims that live forever as queryable context.

This is the moment the substrate stops being theory and starts behaving like capital. The next time we assemble a brief on the same topic, `claim.recall_decisions` will retrieve "Hook 2 won over the other 3" as context for the composer. Sonnet does not need to remember it; the graph hands it over. That is what distinguishes a substrate from a RAG: it does not search for similar documents, it serves typed claims with provenance.

**After a single session**, Roberto's workspace has four productive PlanTemplates, seventeen atomic Operations, sixty some Claims accumulated as compounding context, and a real operating cost of roughly two cents per executed workflow. That is not the important part. The important part is that every one of those Claims will still be here tomorrow, and a year from now, feeding the next workflows we assemble.

*Extension · 2026-05-20 · first external standard absorbed*

## The substrate *absorbs* a community standard

[audience.md](https://github.com/KikoPalomares/audience.md) appeared: an open Markdown standard, written in 2025, for defining audiences so that humans and agents consume them the same way. The question was not whether it replaced the substrate. The question was whether the primitives we already had could absorb it without a rewrite. The short answer is yes. What follows is the evidence.

*What audience.md is (summary)*

A plain Markdown file (`AUDIENCE.md`), versionable in git, with twelve canonical sections: name, summary, primary audiences, jobs to be done, pains, motivations, decision criteria, language and tone, anti goals, evidence vs assumptions, open questions. Optional YAML frontmatter with status, owners, last_reviewed. Designed so that any LLM agent reads it as base context before operating on the audience.

*Mapping to the primitives we already had*

### Zero new primitives. Four clean alignments.

Conceptually equivalent to any other published Artifact, except its lifecycle is "always present" instead of "produced by an Op". It lives in the filesystem, not in R2.

It defines which predicates are valid for talking about the audience (hasJob, hasPain, hasDecisionCriterion). Other workspaces can have different overlays.

This is a perfect match. Evidence with a verifiable source = high confidence Claim + populated source_refs. Assumption = low confidence + empty source_refs. The standard and the substrate describe exactly the same distinction.

It satisfies Rule A: declarative, versionable, durable. Identical philosophy to the workspace manifest.

*The wiring · brief-synthesis-v1 before and after*

### One new step. Zero refactors to the rest of the DAG.

> *[SVG diagram: see the web version]*

*The concrete file · workspaces/ai4managers/AUDIENCE.md*

### The first AUDIENCE.md derived from Roberto's brand blueprint.

```
---
audience_id: ai4managers
status: stable
owners: [roberto@aguirre]
last_reviewed: 2026-05-20
source_doc: basic-memory:/sistema/reference_ai4m_brand_blueprint.md
---

# Audience: AI4Managers

## Summary
Spanish-speaking middle-to-senior managers (40-55) who built consolidated
careers before the AI shift and now sense the rules of their profession changed...

## Jobs To Be Done
- Understand what AI changes in MY specific role this quarter
- Operate AI tools and agents without becoming an engineer
- Preserve professional standing against AI-fluent juniors

## Language And Tone
**Core terms**: operar, desplegar, agentes en producción, manager, battle-tested
**Prohibited terms**: "estamos en la era de…", "imaginá si…", "lab", emojis,
  jerga ingenieril sin traducción (LLM, RAG, embeddings, fine-tuning)

## Anti-Goals
- Beginner-tutorial content
- Engineering deep-dives that require coding fluency
- English-only or US-only case studies

## Evidence
- Roberto's professional transition (Manager Senior → Head of AI Ops Engineering)
- Corporate training engagements (only currently validated monetization)
- LinkedIn outreach with ~207 unique authors, 33 batches

## Assumptions
- LATAM and Spain share enough vocabulary to be a single audience
- Status preservation > upside ambition (untested at scale)

... 4 more sections: Motivations, Decision Criteria, Open Questions, Content Pillars
```

*What changes in Sofia's prompt*

### The prompt now has a precedence rule and a pillar guard.

The pillar guard is the interesting piece. If someone dispatches an Intent to write, say, a basic prompt engineering tutorial, which does not fit any of the four pillars in the brand blueprint, Sofia does not produce the brief: she flags it `PILLAR_MISMATCH` with a reason. The human gate receives it and Roberto decides whether to widen the pillars or discard the piece.

*Backward compatibility · zero breakage*

Workspaces without an `AUDIENCE.md` keep working identically to the day before. The handler returns `{ present: false, doc: null }` and Sofia falls back to the `target_audience` string as she always did. The earlier smoke tests (standup, lead, brief without audience, video) pass without touching anything. The extension is opt in at the workspace level, not a global behavior change.

*The exact delta · what this extension weighs*

### 120 lines inserted. 6 files modified. 5 new files. 6 more tests.

**The architectural reading.** This is the first time an external community standard maps cleanly onto the substrate's primitives without forcing. That confirms a hypothesis we had been assuming: the symbolic model has enough expressive power to absorb community conventions without rewriting them. The next candidate extension is *llms.txt* (Jeremy Howard's standard for AI crawlers to consume docs); if it maps again to Artifact + WorkspaceManifest, the "absorb, don't reinvent" pattern becomes doctrine.

**The operational reading.** The next `produce_artifact / content-brief` Intent dispatched against the AI4Managers workspace will generate briefs aligned with the brand blueprint without anyone having to paste the document into the prompt each time. The thesis, the pillars, the prohibited words, the anti goals: everything is in the graph, and it is applied automatically.

*Chapter twelve · As built · snapshot 2026-06-11 · written for engineering students*

## The architecture *today*, piece by piece

The earlier chapters tell the origin (May 2026: the substrate was born in one session). This chapter takes a snapshot of what exists **today, built and verified in production**: what each component is, what it is for, where it lives, and how it connects. What in May was 16 operations and 28 tests is today 73 operations, 13,776 tests, and a 34 step manual that a browser re-verifies against production. If you are going to steal something from this case, steal the criterion: *every piece exists for an operational reason, none for hype.*

**The system in one sentence:** a stateless app on Vercel declares *intents* to an engine on a Hetzner box; the engine compiles them into typed plans, executes them durably with an LLM whose cost is accounted for step by step, forces a human approval at the end of every plan, and turns every human decision into structured memory queryable with SQL.

Seen from outside, a running plan looks like a graph of steps with one node that queries a vector database. From there comes the easy criticism: *"this is an n8n template with a vector node, and that's it."* It confuses the **artifact** (a DAG at runtime) with the **process that generated it**. n8n is a flow a human draws by hand. Here the flow **is not drawn: it is composed and validated.** Three differences in kind:

Nova (an LLM) **generates** the plan from natural language; the server **formally validates** it against the catalog: refs and edges (`validatePlanAgainstCatalog`), a graph without cycles by Kahn, a cap of 10 steps. A human did not draw this graph; the system derived it and proved it.

Karina, Alexa, Sofía are not processes and have no capability: they are **identity labels**. Capability lives in the catalog's operations, and Nova assigns the actor from there; it never reads the squad. It is the inversion no node based flow has: **the team is derived from the work, not the other way around.**

Of 73 operations, **55 are deterministic code handlers**; the LLM only enters in the 18 that *compose* or *judge* text. And every plan ends in a human gate with `fallback:'fail'`: the model has no path to skip it. It is neither "the LLM does everything" nor "everything hardcoded".

The sophistication is not where critics look (that a DAG exists, that a vector exists) but where they do not: the validated composition, the catalog↔persona inversion, and the structural gate. Four real architects (Cherny, Embiricos, Chase, Majors) reviewed this as built system: [minutes of the session →](master-arq-asbuilt.html)

### The layer map: who talks to whom

- **Browser** Hydrated SvelteKit 5 (runes) · 3D office Threlte/Three.js · custom SSE parser
- **SvelteKit server · Vercel** 13 product pages + 15 endpoints · fail closed gates per request · server first auth
- **InsForge (BaaS)** identity: auth.users + JSONB profile `app_state` · `user_access` (gate) · `magic_links` (one time)
- **Resend** Transactional SMTP (magic links)
- **Cloudflare → nginx** TLS · exposes ONLY health, workspaces/*, approvals, intents · `proxy_buffering off` for SSE · Inngest is NOT public
- **Hono on Bun · systemd :4000** 10 routes · rate limits · plan validation against the catalog · Nova · `idleTimeout: 120`
- **Inngest OSS** durable execution · human gate `step.waitForEvent` ≤24h · survives restarts
- **LLM adapter** Claude Code headless (CLI, $0 marginal) · API fallback · only `text_delta` reaches the user
- **Postgres 16 + pgvector/pgvectorscale** 77 substrate tables · DiskANN 3072d on claims and artifacts · 32 monthly partitions
- **Langfuse v3** web + worker + ClickHouse + MinIO + Redis · trace and cost by trace_id
- **Crons + backups** pg_dump nightly 03:00 UTC → `~/backups/substrate/` (+optional R2) · standup digest 07:30 with CLI preflight

Three compute domains, each where it fits best: the stateless in Vercel (scales on its own, stores nothing), the stateful on a box with disk (Postgres, Inngest, Langfuse: 3 Docker Compose stacks + one systemd), and identity in a separate BaaS, decoupled from the substrate by design. Total memory observed on the engine side: ~1.7 GB of 15 available.

### The pieces, one by one

#### SvelteKit 5 + Vercel consumer app

**What it is.** A full stack app with runes (compiled reactivity) served serverless. 13 product pages (welcome, onboarding, office, briefing, library, activity, outputs, squads, hire, squad-proposal, discover, deep-dive, share; the public demo, the OAuth callback, and the legal pages are separate), each with its own `+page.server.ts` that loads real data from the engine, and degrades to labeled sample content if the engine does not respond.

**What it is for.** It is the face of the product: everything the founder sees and touches. Stateless on purpose: it stores nothing, so Vercel can create and destroy instances without losing state.
`apps/web/src/routes/ · adapter-vercel · maxDuration 90s en chat y compose`

#### The 3D office Threlte + Three.js

**What it is.** A voxel scene rendered in the browser: 16 agents in 3 squad zones, 12 desks, with the layout computed by a pure function (tested with vitest, without touching WebGL).

**What it is for.** It turns "you have agents working" into something you can see. The office recap ("N shipped · M await review") comes from an engine endpoint that aggregates real traces from the last 24h.
`apps/web/src/lib/scenes/ · OfficeScene, VoxelAgent, SquadZones, CameraRig`

#### ChatDrawer + streaming SSE parser

**What it is.** The chat with each agent. A custom SSE parser (~60 lines, pure, tested) paints each fragment of the response in a transient bubble; when the drawer opens with an empty session, it hydrates the thread persisted in the engine.

**What it is for.** Talking to an agent with no process waiting: the agent's "continuity" lives in its memory (briefing + jobs + thread), which the engine assembles at the moment of the message. If the stream is cut before the first fragment, it degrades to the classic path with a single retry.
`lib/components/chat/ChatDrawer.svelte · lib/chat/sse.ts · lib/chat/session.ts`

#### NovaModal + SkillLaunchModal single door

**What it is.** The "ask Nova" UI: a natural language textarea and a modal that shows the outcome: a match with an existing SuperSkill, a newly composed plan (with estimated cost and steps in plain language), or an honest "I can't". From there the plan is saved under its own name.

**What it is for.** The user never navigates a technical catalog: they ask. And what Nova composes can be promoted to a relaunchable SuperSkill from the library.
`lib/components/library/NovaModal.svelte · SkillLaunchModal.svelte`

#### InsForge identity BaaS

**What it is.** A backend as a service that provides `auth.users` with a passthrough JSONB profile, plus two tables of its own: `user_access` (a boolean private beta gate, fail closed) and `magic_links` (CSPRNG one time tokens, 15 min, atomic consumption with `UPDATE … WHERE consumed_at IS NULL`).

**What it is for.** All identity, separated from the substrate by design. The user's app state (squad, onboarding, installed SuperSkills) lives in `profile.app_state`: the profile IS the consumer's database.
`migrations/2026…create-user-access.sql · create-magic-links.sql · SDK @insforge/sdk`

#### hooks.server.ts the per request guardian

**What it is.** The SvelteKit hook that runs on EVERY server request: it reads the httpOnly cookie, renews the access token with the refresh token if it expired (~15 min), and queries `user_access` with the service key.

**What it is for.** Truly fail closed: an error reading the gate = not authorized. The bypass for E2E tests exists but is doubly gated (`dev && CI`); in a production build that branch is dead code.
`apps/web/src/hooks.server.ts`

#### Server first auth the refresh token lesson

**What it is.** Login, Google OAuth, and magic links are exchanged *on the server* (`client_type=server`), never in the browser. Reason: the browser SDK does the PKCE exchange in its constructor and never exposes the refresh token; without it, sessions died at 15 minutes.

**What it is for.** 30 day sessions with silent renewal, and tokens that live only in an httpOnly/secure cookie.
`lib/server/auth.ts · routes/api/auth/* · routes/auth/callback`

#### /api/user/state state with discipline

**What it is.** The only endpoint that writes `app_state`: it does a read, merge, and write of the full state, resanitizes fields with an explicit pick (junk does not persist), and rejects bodies >64KB. On the client side, a *write queue* serializes the POSTs (two in flight writes would overwrite each other) with an optimistic update and a compare and swap revert.

**What it is for.** Multi device without a state server: you create a squad in one browser and see it in another. localStorage stays as a write through cache, with precedence profile > cache > derivation.
`routes/api/user/state/+server.ts · lib/stores/userState.ts · lib/squads/store.ts`

#### Cloudflare → nginx minimal surface

**What it is.** TLS with a Cloudflare origin certificate and a single nginx site that proxies 443 → 127.0.0.1:4000, exposing only `/health`, `/api/workspaces/*`, `/api/approvals`, and `/api/intents`. The Inngest endpoint does not exist for the internet. For the chat SSE: `proxy_buffering off` and a 120s read timeout.

**What it is for.** So the engine has exactly the surface the app needs and not one path more. Port 4000 is closed to the public by firewall; only loopback and the Docker bridge reach it.
`substrate-infra/nginx/api-substrate.digitalhubassist.ai.conf`

#### Bearer + rate limiting middleware

**What it is.** A server to server bearer token compared in constant time (anti timing attack), and an in memory sliding window rate limiter: chat 10/min, compose 4/min, intents 10/min. It is mounted *after* the bearer check (a 401 does not consume quota) and answers 429 with `Retry-After`.

**What it is for.** The LLM is the expensive resource; the limits protect the session from an accidental loop or an eager script. The browser never sees this token: only the server side app uses it.
`apps/api/src/middleware/bearer-auth.ts · rate-limit.ts`

#### Hono on Bun systemd :4000

**What it is.** A single HTTP app with 10 route files: `POST /api/intents`, `POST /api/approvals`, and under `/api/workspaces/:id/` outputs, activity, brief (GET/PUT), manifest, chat (+history), compose (+launch/discard), superskills (+launch). It runs as a systemd unit with restart on failure and logs to file.

**What it is for.** It is the only entry point to the substrate. A detail with a scar: `idleTimeout: 120` in the server export. Bun's default (10s) killed the SSE when the model reasoned silently for more than 10s. The living manual caught it, not a unit test.
`apps/api/src/index.ts · routes/ · .runtime/{stdout,stderr}.log`

#### The catalog: 73 Operations the substitution seam

**What it is.** Seventy three typed, versioned refs spread across 33 domains (`trace.query`, `claim.recall_decisions`, `text.compose_brief`, `video.compose`…); that is 70 ids, because three coexist in two live versions. The largest domain is `research`, with 17. All of them have a runtime handler except two, `human_gate.approve` and `human_task`, which lack one *on purpose*: the executor itself runs them by suspending the workflow.

**What it is for.** The operation is the unit with a contract (typed inputs → outputs + cost) and the seam where any volatile dependency would be quarantined: an agent framework fits *inside* an operation; the reverse does not.
`packages/substrate-spec/src/catalog/ · apps/api/src/inngest/operations/`

#### Compiler + 23 PlanTemplates + 12 evaluators Intent → Plan

**What it is.** A declared Intent is compiled into a Plan: a topologically sorted DAG of steps where each step references a catalog operation. The plans Nova composes go through 4 layer server side validation (each operation exists in the catalog, the graph has no cycles (Kahn), a cap of 10 steps, and the final approval is added by the system); every launch, whether factory, composed, or saved, revalidates against the catalog and structurally requires the terminal human gate. Evaluators emit verdicts per step.

**What it is for.** So that "the model decided" is never the explanation for an execution: what runs is a typed artifact that the server validated, with an estimated cost computed server side (the model cannot lie about it).
`apps/api/src/substrate/plans.ts · packages/substrate-spec/`

#### Nova (compose) one call, three outcomes

**What it is.** A single query to the model decides: MATCH (the request corresponds to an existing SuperSkill, validated server side against the real list; the model cannot invent ids), COMPOSE (builds a new plan using ONLY catalog operations), or CANNOT (says so honestly and the request is recorded as demand).

**What it is for.** It is the only place in the system where the LLM "designs", and that is why it is the most closely watched: what it proposes goes through its 4 layer validation and through the same structural gate as everything else. Every request is stored in `plan_drafts` as telemetry: the "couldn't do it" entries prioritize what to build.
`apps/api/src/substrate/nova-compose.ts · routes/compose.ts`

#### SuperSkills + launch-plan promoted plans

**What it is.** A composed plan that people liked is promoted under its own name into the workspace catalog: it is revalidated against the catalog BEFORE the insert, and a partial unique index in the DB makes a double promote impossible (exactly one wins, the other gets a 409). Launching uses a helper shared with the factory plans: `assertHumanGatePresent` structurally verifies ≥1 terminal human gate with fallback 'fail'.

**What it is for.** So that what the user creates by asking has the same guarantees as what the team built. The relaunch form is derived mechanically from the variables the plan itself declares.
`apps/api/src/substrate/superskills.ts · launch-plan.ts · routes/superskills.ts`

#### LLM adapter Max CLI + API fallback

**What it is.** A custom adapter with two backends: Claude Code headless via CLI (a Max subscription session, $0 marginal cost) and the official API as a fallback. For streaming it parses NDJSON line by line and emits ONLY `text_delta`; the model's internal reasoning never reaches the user; the final `result` line wins over what was accumulated.

**What it is for.** Separating architecture from stage: the beta runs at zero cost, and since every step records its notional cost in dollars, the unit economics of switching to the API are known TODAY ($0.013-0.022 per digest). Turning on the API is configuration, not rearchitecture.
`apps/api/src/inngest/llm.ts · LLM_PROVIDER=claude-cli`

#### Inngest OSS durable execution

**What it is.** The self hosted durable runtime for events, crons, and operational retries. `handle-intent-declared` selects v2 and starts a run over the compiled graph; `worker-orquestacion` ticks the worker that consumes `work_items`; the sweeps recover expired leases and gates; `materialize-manifest` keeps its cron. `execute-plan` remains only as the v1 executor being deprecated and an anti zombie canary.

**What it is for.** Inngest makes delivery and housekeeping durable, but it is no longer the authority on DAG progress. That truth lives in Postgres: `runs` + `run_events`, folded by the reducer, and the transactional queue the v2 worker processes. A deploy can interrupt a tick; it cannot erase what the run has already decided.
`apps/api/src/inngest/functions/ · apps/api/src/orquestacion/ · substrate-infra/inngest/ · :8288 (no público)`

#### Postgres 16 + pgvector/pgvectorscale the physical substrate

**What it is.** A dedicated instance (Docker, :5433) with 77 logical tables and 32 monthly partitions. The two highest volume tables, `traces` and `step_executions`, are partitioned by month. The embeddings (3072d, text-embedding-3-large) are a column on `claims` and `artifacts` with a StreamingDiskANN index; HNSW tops out at 2000 dimensions, which is why pgvectorscale.

**What it is for.** Semantic memory and typed facts in the SAME store: a JOIN between vector similarity and structured claims, with no dual write against an external vector DB. Consistency comes free because there is a single database.
`db/substrate/migrations/0001-0007 · substrate-infra/postgres/`

#### The provenance graph mandatory lineage

**What it is.** The chain `intents → plans (steps, edges) → traces → step_executions → artifacts / claims`, plus `lineage_edges` with a recursive view (max depth 12, cycle guard). Hard rule: every claim and artifact MUST have provenance `{trace_id, step_id}`.

**What it is for.** Native audit: for any claim in the system you can answer "where did this come from?" by walking the graph: which trace emitted it, which step, which human confirmed it.
`db/substrate/migrations/0001_init.sql · lineage_upstream view`

#### workspace_manifests Rule A

**What it is.** A navigable snapshot per workspace (squad, active intents, recent artifacts, decisions, navigation index) materialized every night and on demand by event. Since migration 0006, the `manifest` JSONB column is the source of truth.

**What it is for.** So an agent starts out knowing where it stands without listing half the database: it reads ONE record. It is a derived cache; it can be rebuilt entirely from the graph.
`apps/api/src/substrate/manifests.ts · inngest/functions/materialize-manifest.ts`

#### Langfuse v3 LLM observability

**What it is.** A self hosted stack of 6 services (web, worker, ClickHouse, MinIO, Redis, its own Postgres) that traces every LLM call with latency, tokens, and cost, correlated by the substrate's trace_id. With a scar of its own: ClickHouse capped at 2 CPUs after a runaway that pinned 7.7 cores for 19 hours.

**What it is for.** Answering "what exactly did the model do in this step and how much did it cost?" with one click. The only Redis in the system lives inside this stack; it is not on the app's path.
`substrate-infra/langfuse/ · :3030`

#### Crons + backups the nightly routine

**What it is.** Two crons: a nightly backup (03:00 UTC, gzip `pg_dump` into `~/backups/substrate/`, 30 day retention, optional push to R2) and a daily standup digest (07:30 UTC, declares a real Intent through the public API with a bearer, with a preflight that verifies the CLI's health before firing).

**What it is for.** The backup is the answer to the catastrophic failure domain (RPO ≤24h). The digest is the product's heartbeat: the system uses itself every day. The dumps grow with the system: 680K in May, 1.8M today.
`substrate-infra/scripts/backup-substrate-db.sh · standup-digest-daily.sh`

#### Verification as a product 13,776 tests + living manual

**What it is.** Three rings: 13,333 unit and integration tests (4,102 from the app, 9,058 from the engine against a real Postgres, 173 from the spec), 443 E2E/visual tests with Playwright spread over 81 files (full journeys + screenshot regression), and the outer ring: [a 34 step manual](manual-agentsquad/) that a real browser reexecutes against production before every demo, with an ephemeral user and surgical cleanup of the workspace.

**What it is for.** The outer ring catches what the inner ones do not see: the `idleTimeout` bug (SSE dead after 10s of silence) passed every unit and E2E test; the manual found it. If a step breaks in production, its seal stops saying VERIFIED.
`tools/manual-walkthrough/ · apps/web/tests/ · apps/api/src/**/*.test.ts`

### Anatomy of three requests: the system in motion

The static architecture is half the story. These are the three paths that teach the most, step by step:

*Flow 1 · Running a SuperSkill (the full cycle)*

01. The founder taps "Run" in the library. The browser sends ONLY `{workflowId, input}`; nothing else travels.
02. The app's proxy validates session + gate, validates the input against the closed catalog, and builds the full Intent server side: `declared_by = human:<email>`. POST to the engine with a bearer.
03. The engine compiles the Intent into a Plan (typed DAG) and validates it against the catalog; `assertHumanGatePresent` structurally verifies the terminal human gate. If the plan does not end in human approval, no execution is possible.
04. Inngest walks the DAG: each step dispatches its operation and records input/output snapshots, the evaluator's verdict, and the cost in dollars. If the process restarts midway, the execution continues where it was.
05. The last step produces a `pending_review` Artifact and the workflow SUSPENDS (`step.waitForEvent`, up to 24h). There is no polling: the process sleeps.
06. The founder approves with a comment in /outputs. The event resumes the workflow: artifact → `approved` and the decision materializes as typed Claims (`approvedBy`, `approvalComment`…) that the next plans will query via `claim.recall_decisions`. If nobody decides in 24h: terminal state `expired`, no orphans.

*Flow 2 · A question to the chat, streaming*

01. The drawer opens with an empty session → it hydrates the latest thread from the engine (fail soft: if it does not respond, the chat opens empty and still works). Sending continues the same `conversation_id`.
02. The engine persists the user's message, assembles the agent AT THAT INSTANT (persona + briefing + latest jobs with approval comments + thread), and launches the CLI in stream mode.
03. The NDJSON parser reemits ONLY `text_delta` as SSE events. nginx does not buffer (`proxy_buffering off`); the Vercel function pipes the body without accumulating; the browser paints each delta in the bubble.
04. The teaching scar: Bun's default cut every silent connection >10s, and the model reasons in silence (its thinking deltas are filtered out). Short questions got through; the "real work" ones died and degraded to the classic path. Fix: `idleTimeout: 120`. The living manual caught it in production.
05. The `done` event brings the canonical reply; the transient bubble disappears and the reply is persisted ONLY if the stream completed. No half answers are left in the history.

*Flow 3 · Nova composes, the user saves it*

01. The natural language request goes to the engine. ONE call to the model decides: a match with an existing SuperSkill, a new plan, or "I can't"? The match is validated against the workspace's real list; an invented id does not exist.
02. If it composes: the plan uses ONLY catalog operations and passes Nova's 4 validation layers (existence, no cycles, step cap, final approval). The server calculates the estimated cost from the catalog. The user sees the steps in plain language and the price.
03. "Save as SuperSkill": the plan is revalidated and inserted into the workspace catalog. Double saving is impossible: a partial unique index in the DB, and exactly one wins the race.
04. The card appears in "Your SuperSkills", read from the engine, fail soft. Nova sees it in future requests (≤20 customs in its prompt) and can offer it instead of composing from scratch.
05. Relaunching goes through the SAME helper as the factory plans: `assertHumanGatePresent`; without a terminal human gate, there is no launch. The guarantees do not depend on who created the plan.

### The data model: 15 tables, each with its why

| Table | What it stores | The why |
|---|---|---|
| intents | What who wants to achieve, in which workspace | The audited starting point of EVERY execution; supports a tree (parent_intent_id) |
| plans + steps + plan_edges | The executable DAG compiled from the Intent | Versioned per intent; each step carries operation_ref, retry policy, timeout, and an optional human_gate |
| operations | The catalog: metadata for each operation, versioned | Deprecate without deleting; the plan references `id@version` |
| traces | Each concrete execution of a plan (state, real cost, verdict) | Partitioned by month; entry point to the lineage |
| step_executions | Each executed step: input/output snapshots, error, verdict, cost | The fine grained audit; partitioned the same as traces |
| claims | Typed facts (subject · predicate · object) + provenance + confidence + 3072d embedding | The team's memory; recallable by SQL and by similarity (DiskANN) |
| artifacts | What was produced (docs, videos, digests…) content addressable (sha256), with a status machine | pending_review → approved/rejected/expired; dedup by content |
| lineage_edges | Provenance edges between claims and artifacts | Recursive view (depth 12, cycle guard): "where did this come from?" |
| evaluators | Catalog of how to evaluate (deterministic / llm_judge / human) with a threshold | The per step verdicts come from here |
| ontology_overlays | Per vertical relabels over the same graph | Workspace A says "video reel", workspace B says "blueprint": same primitives |
| workspace_manifests | The navigable snapshot per workspace (Rule A) | Derived cache, rebuildable; manifest JSONB column as SSOT |
| chat_messages | The founder ↔ agent thread per conversation | The engine reads the last 20 to build the prompt; the app hydrates on reload |
| plan_drafts | Each request to Nova with its outcome (proposed/matched/rejected/launched) | Demand telemetry: the "couldn't do it" entries prioritize the roadmap |
| superskills | Promoted plans with a name, per workspace | Partial unique index against double promote; archive instead of delete |
| + in InsForge (identity, separate from the substrate): `user_access` (fail closed private beta gate) · `magic_links` (one time tokens with atomic consumption) · `profile.app_state` (the consumer's state) |  |  |

A detail a student should notice: `workspace_id` is on every workspace *data* table since migration 0001 (intents, traces, claims, artifacts, lineage_edges, manifests, chat, drafts, superskills); the definition tables (plans, steps) inherit it through their intent, and the catalog tables (operations, evaluators, overlays) are global by design. The data model is multi tenant from day one; the isolation still pending is in the routing layer, not in the data.

### Seven decisions to steal

### The as built numbers

Measured against the repo on **2026-09-03**, not estimated: the catalog is counted by importing `OPERATION_CATALOG`, and the tests by running the suites. Any of these numbers that goes stale is a bug in the document.

| | |
|---|---|
| **73** | Operations |
| **23** | PlanTemplates |
| **12** | Evaluators |
| **77** | tables (+32 partitions) |
| **13,776** | tests (13,333 unit + 443 E2E) |
| **34** | steps verified in prod |
| **~1.7GB** | engine RAM (of 15) |
| **$0.013-0.022** | per digest (notional) |

*What is NOT there yet: the other half of honesty*

Multi tenant routing (the data model is ready; the beta serves one workspace with a founder gate) · high availability and a formalized RTO (there is run durability and a nightly backup, but no replica and no documented restore drill) · proactive alerting with SLOs (there is observability, but nobody gets woken up at 3am) · offline regression of evals (the human gate already accumulates the labeled dataset; the harness in CI is missing). All four are acknowledged, classified, and committed to; the detail lives in the [architecture red team](comite-red-team.html).

**To close the case study.** The foundation of this system is brands any student recognizes: Postgres, nginx, Vercel, Cloudflare, Anthropic, Inngest. What is custom is a thin, typed, tested layer. The question that ordered every decision was not "what technology is in fashion?" but *"what mature, cheap thing covers this without painting us into a corner?"*, and the only truly original bet, the structured memory of human decisions, is exactly the piece no framework sold. The full tour of the product, with every step verified against production, lives in the [operator's manual](manual-agentsquad/).

*Chapter thirteen · The Board of Architects responds*

## The Board *responds*

Four real architects of coding agents took the questions left by the students and a parent, and answer them from their own lens. **Boris Cherny** (Claude Code · Anthropic), **Alexander Embiricos** (Codex · OpenAI), **Harrison Chase** (LangChain/LangGraph), and **Charity Majors** (Honeycomb). Quotes marked ⚠ are recaps of public talks (not a literal transcript); those marked ✓ are verbatim from their own source.

- **Boris Cherny** · do the simple thing
 IRIS is scaffolding for a bottleneck this system does not have. Mirroring everything to RAM solves the latency of *high traffic synchronous load*: hundreds of thousands of users, 10-15 tool calls per question. Agent Squad's load is the opposite: few, deep, asynchronous workflows; a digest takes minutes by design. When you add a layer, ask yourself whether it survives the next model or whether you are adding it because today's model falls short.
 ⚠ recap: "scaffolding gains get wiped out by the next model"
- **Charity Majors** · what breaks at 3am?
 The cost IRIS *charges* you is the one that hurts most in production: a second copy of your data via CDC is a second system to keep consistent, and consistency between two stores is exactly what fails on a Sunday at 3am, without you having anticipated it. Do not operate what you do not need to operate.
- **Harrison Chase** · context engineering
 IRIS is a *context* solution for high traffic. Here the context is controlled by the catalog and the typed claims: you know exactly what enters the LLM at each step, which is the truly hard problem. An honest nuance: if a high traffic synchronous agent ever shows up, the IRIS pattern fits inside an operation. Not today.
 ✓ blog: "make sure the LLM has the appropriate context at each step"
- **Alexander Embiricos** · you were right
 The intuition is correct and has already been acted on: there is an initial conversation where the system discovers *what work* you need, proposes the capabilities, and derives the team from there. The "3" was a presentation default, never a restriction. What comes next is your second half, the last mile: the agent producing with more autonomy, not the human driving step by step.
 ⚠ recap: "you're delegating to it rather than pairing with it"
- **Boris Cherny** · people follow the catalog
 The squad was never the center of the design. The people are `identity labels`; capability lives in the operations catalog. The planner composes from the catalog and never reads the squad, so the number of people *emerges from the work* rather than being fixed in advance. Hardcoding the size would have been the mistake the student feared; the system avoids it by construction.
- **Harrison Chase** · validated composition
 What you see is neither hardcoding nor total autonomy: it is a router/state machine that *generates* the plan from natural language and *validates* it against a catalog, with guardrails. The team is a byproduct of which operations entered the plan. That is the difference between drawing a flow by hand and deriving it.
- **Boris Cherny** Understand *the model* and how to give it tools and a goal, not marry a fashionable framework that the next model makes obsolete. The durable skill is knowing how much scaffolding to put in and how much to delete.
- **Alexander Embiricos** Learn to *direct and delegate* to agents. The bottleneck of the future is not the model's capability, it is the human's speed at assigning and reviewing. Whoever can orchestrate several agents at once performs ten times better.
- **Harrison Chase** It is still *systems engineering*. The hard work of agents is not the model: it is controlling what information enters each step. Systems fundamentals + "context engineering" do not expire with the next release.
- **Charity Majors** Learn to *operate* systems in production: observability, "how do I know this works and how do I fix it when it fails?". AI systems are nondeterministic; understanding how they behave when they break is a skill AI will not automate soon.
- **Charity Majors** · the best seat
 In the *verification ring*: the robot that walks the entire product against production every night. To verify a single step you have to understand ALL the layers that cross it: screen, server, engine, database. It is the only seat from which you see the whole map, and with zero risk: it uses test users that self destruct. Test in prod, from day one.
 ✓ "Nines don't matter if users aren't happy"
- **Alexander Embiricos** The practitioner *is* the system's use case: a worker with energy and no formed judgment yet, just like the model. The same guardrails that contain the AI contain them, so they can do real work from week one, which is the only way to learn.
- **Harrison Chase** After the ring, give them an *Operation of their own* from the catalog: a deterministic piece from end to end. There they learn to control exactly what goes in and what comes out of a step, the core muscle.
- **First, the facts** · what Hermes Agent is (verified)
 An open source framework from **Nous Research** (Feb 2026): a *personal*, self hosted agent that lives in your messengers (CLI, Telegram, Discord, Slack), remembers across sessions, and *improves itself*: after a complex task it writes a reusable "skill" without asking permission, and it connects to 200+ models. Its guardrails are *infrastructure* ones (container hardening, read only root), not approval of its actions. The question is not idle: it is one of the fastest growing tools of 2026. [hermes-agent.org](https://hermes-agent.org)
- **Alexander Embiricos** · it is my thesis taken to the max
 Hermes takes autonomy all the way: it improves itself, runs long, takes the human out of the loop. For a *personal* assistant that is exactly what you want: the human is the bottleneck, remove them. Agent Squad puts in a human checkpoint on purpose, because its effects land on *someone else's business*: a brief that goes out to a client, not a script you run on your own machine. If your case is personal and tolerates autonomy, Hermes fits. If someone has to sign off on every deliverable, that checkpoint is not bureaucracy.
 ⚠ recap: "the bottleneck is the human, not the model"
- **Charity Majors** · where is the guardrail?
 The most honest difference: Hermes puts the guardrail in the *container* (so the agent does not wreck your machine); Agent Squad puts it in the *action* (nothing with an effect happens without a signature) and adds provenance and SLOs. An agent that learns on its own and acts on its own in your messengers: how do you know what it learned, how do you audit it when it does something strange at 3am? For personal use, that opacity is *your* risk, and acceptable. For work that lands on clients, it is not.
 ✓ "Nines don't matter if users aren't happy"
- **Harrison Chase** · same idea, different governance
 Hermes auto creates skills (*emergent* orchestration): flexible for varied personal tasks. Agent Squad derives a typed graph validated against a catalog (*explicit* orchestration): controllable for repeatable, auditable business flows. And a point of humility: Hermes's "closed learning loop" and our *promoted SuperSkills* are the same idea, capturing a procedure to reuse it. The difference is who pulls the trigger: Hermes does it alone; we do it with a human who promotes.
 ✓ blog: "control exactly what goes into the LLM at each step"
- **Boris Cherny** · opposite bets
 Hermes is model agnostic (200+ models) and ships a harness: memory, learning loop, 40+ tools. That is optionality and features, and also scaffolding the next model may make unnecessary. Agent Squad bets on a single frontier model and puts in minimal scaffolding + hard guardrails. Neither is "better": they are opposite bets. The question that matters is not which has more, but which of the two bets serves *your* problem.
 ⚠ recap: "bet on the model, not the scaffolding"

*The journey continues · August 2026*

## From *noise* to the audience

The research engine started by measuring demand: what is searched, what is asked, what gap remains between what people ask for and what exists. It ended up understanding the person who is watching: who they are, what moves them, what holds them back, and the exact words of their pain.

The substrate's market research was already deep. It measured the **compound demand** of each opportunity along three separate dimensions, never one disguised as another: the search for the angle, the market for the topic, and the social backing from Reddit and YouTube comments, attributed against a closed table so the model never writes a number it did not see. A dedicated pass tied down the evidence, and a ranking by Pareto fronts ordered the opportunities without adding apples to oranges.

And then a founder doubted a zero. The report said there were no valuable comments on YouTube, and he did not believe it: *"I manually find cases that are not praise"*. The doubt, insisted on four times, uncovered real layers. First, four defects in how the engine **discovered** the videos: it searched for generic advice ("how to grow") and not for the problem videos ("why isn't my channel growing") where the audience asks questions. And a deeper finding: the lexical classifier could not tell a problem from a curiosity, because the difference was semantic; the clean signal was in another layer, in the provenance of the search, not in the title.

But the bigger discovery came with a video the founder brought in by hand: an instructor's masterclass, whose comments were personal stories. There the value was not demand. It was **audience intelligence**: the profile of who is watching, their motivations, their barriers, and the gaps they reveal by telling their story. The engine was chasing it without seeing it. That is how a new pass was born, r14, which reads the comments the research already sampled and extracts five honest things against a closed table: **profile, motivations, barriers, gaps, and pain phrases** as a literal verbatim quote, ready for the script.

| | |
|---|---|
| **3** | New waves |
| **5** | Languages read |
| **100** | Comments per run |
| **r14** | Audience pass |
| **v10** | Template in prod |

In a real production run, in five languages, the report wrote down what used to be lost: a **70 year old creator who uses AI as an assistant**, one who identifies as **neurodivergent** and loses the thread if they preproduce several videos, one who writes in French *"j'espère que je vais y arriver avant que la maladie m'emporte"*. Real words from real people, written into the brief, ready to become content. The zero the founder did not believe was never the end: it was the tip of the thread.

*Sovereignty and guardian · August 2026*

## The *guardian* between the engine and production

An engine that evolves fast needs something that stops it from carrying a broken change all the way to users. Not trust in intent: a guardian that only lets through what has proven it works.

The substrate stopped depending on third parties for its own integration. The code and the CI became **sovereign**: they live on a box of our own, without asking anyone's permission. And with that move came the question that really matters: when the engine changes, what decides whether that change ships to production or not?

The end to end test suite had spent days marked "red" and outside that decision. On a closer look, something different appeared: it was not broken, it had been **green for five nights in a row**. The traffic light was lying. What the gate's comment declared was corrected, and the real debt was left in plain view, the only one still to pay: for that suite to once again **stop a bad deploy**, not merely watch it go by.

The finding that organized everything was this: there is not one deployment, there are **two**. The one for the visible face and the one for the engine, each along its own path. And they are not validated the same way. The end to end tests run against a simulator of the substrate, so they tell the truth about the interface but not about the engine, which has its own suite. Treating them as one would have made a stumble in the face stop the engine for no reason. Each guardian watches its own door.

| | |
|---|---|
| **2** | Distinct deployments |
| **6** | Test shards |
| **5** | Nights in green |
| **✓** | Bite: red blocks |
| **0** | External dependency |

With the door made clear, the face's suite began to block its deploy: the light interface tests on every change, and the heavy tests of the live scene when the change touches that scene. The engine continues on its own path, validated by its own. The substrate is no longer just built fast: it is published without a broken change slipping through, and that guarantee has been proven by hand, bite by bite.

And the guardian was refined two more times. It stopped **exposing its key**: the token it publishes with no longer travels on the command line, where any process on the server could read it, but in the environment, out of sight. And it learned **not to freeze**: once the publishing tool hung on a network call and stayed stuck until the turn's cap, forty five minutes, until it had to be killed by hand. Now every publishing step has a hard time limit and retries on its own, so a hang is cut off within a few minutes instead of jamming everything. A guardian that neither leaves its key in plain sight nor falls asleep at the door.

*Chapter fourteen · Replacing the engine*

## From two engines to *one*

For weeks, two authorities coexisted over a run. Not today: production runs v2 at 100%. Inngest keeps the durability of events and crons; the progress of the work belongs to the event sourced engine.

The original engine (v1) lived in `inngest/functions/execute-plan.ts`. It received `plan.compiled`, walked the DAG of `plans/steps`, and kept a workflow suspended with `step.waitForEvent` when it found a human gate. It worked, left useful traces, and proved the thesis. It also concentrated too much authority inside a function that is hard to reconstruct, inspect, and replace.

v2 changed the unit of truth. The `compilador` turns the template into a versioned graph of `graph_nodes`. A run lives in `runs`; what happened to it lives, in order, in `run_events`. A pure `reducer` decides transitions and effects; `aplicar-evento` persists them transactionally; the worker claims leases on `work_items`, executes operations, and returns new events. Derived state, a reproducible log, an explicit queue. That architecture won because it can be explained after a crash, not only while everything is healthy.

#### Engine v1 executor being deprecated

**What it did.** `execute-plan` walked the Plan, wrote steps and traces, retried operations, and suspended the workflow at the gates with `step.waitForEvent`.

**Why it is going away.** Its effective state was split between Inngest's history and the substrate's projections. Resuming, rebuilding, or changing the engine required understanding a live execution, not folding a stable log.
`apps/api/src/inngest/functions/execute-plan.ts · v1, inalcanzable para lanzamientos nuevos`

#### Engine v2 100% of production

**What it does.** Compiles templates into immutable graphs, records every transition as an event, and moves work through a queue with leases. The reducer does no I/O: it declares. `aplicar-evento` persists. The worker executes.

**What Inngest keeps.** Durable event entry, crons, operational retries, and the tick that ticks the worker. Inngest is still in the system; it stopped being the place where a run's state machine hides.
`apps/api/src/orquestacion/ · graph_nodes · runs · run_events · work_items`

#### Two phase gate own durable flow

**There is no longer a `waitForEvent` holding up the Plan.** The `ACTIVAR_GATE` effect transactionally creates a record in `artifact_approval_flows`: the deliverable appears for review, the person approves and, when the protocol requires it, chooses an option; the executor acknowledges receipt, and only then does the flow complete.

`consumirHandoffV2` applies the decision to the run and chains the successors. The lock order is an invariant, not a recommendation: `runs → artifact_approval_flows → work_items`. It prevents approval, timeout, and worker from closing the same gate in incompatible orders.
`migrations 0073-0075 · reducer.ts · aplicar-evento.ts · aprobar-gate.ts`

#### Timeout, authority, and quality fail closed

Each flow freezes `deadline_at`, a fallback policy, and `required_role`. The `barrido-gates-vencidos` cron claims only the state it observed, via CAS: `state = fase AND deadline_at <= now()`. An approval that won the race cannot be overwritten by the timeout.

The gate is not the only barrier. An evaluator configured as blocking emits a hard stop verdict: if it fails, downstream is not dispatched. The run does not turn a bad output into later work just because the worker is still alive.
`artifact_approval_flows.deadline_at · barrido-gates-vencidos · evaluator gating`

W1
 **v1 became unreachable.** `seleccionarMotor` never returns v1: a flag other than `active + v2` fails closed. The handler and `launchCompiledPlan` do not fall back either. An architecture guard requires zero emitters of `plan.compiled`. If the old executor receives one, it emits the canary `orquestacion.v1_zombie_awakened` and triggers an SLO alert.
 prod · closed



 W2
 **The kill switch started telling the truth.** `orchestration_state = active|paused` controls admission and is separate from `orchestration_engine`. Pausing does not change engines. In addition, `WORKER_ORQUESTACION` is on by default and is set to `on` in production.
 prod · closed



 W3
 **v2 stopped depending on the file we are going to delete.** The shared helpers moved out of `execute-plan.ts` into `substrate/`. The old executor is no longer an accidental library of the new engine.
 prod · closed



 →
 **What comes next is boring on purpose:** a soak period observing zero v1 activity, deleting `execute-plan.ts`, and then cleaning up legacy events, columns, and tables from the schema. First prove the corpse is not breathing; then remove it.
 next wave



 D The two scars


 The escape hatch that was a lie
 “Go back to v1” sounded reversible, but it was not. v1 could not resume a v2 run, was not exercised with real traffic, and executed the template, not the compiled graph that v2 had authorized. Flipping the flag did not recover a run: it created another interpretation of the work.



 The switched off worker that looked like safety
 With two engines, turning off the v2 worker could look like a precaution. With v2 as the only engine it is a footgun: it accepts runs, leaves `work_items` in `ready`, and nothing advances. That is why the default is now `on`; turning it off requires an explicit, observable decision.



 The three operational resets obey the same conclusion: they normalize routing to `orchestration_state='active'` and `orchestration_engine='v2'` without touching `enabled`. Resetting progress does not revoke permission or resurrect a retired engine.






*Chapter fifteen · The current state · surveyed on 2026-10-24, updated on 2026-10-26*

## Today · *2026-10-24*

The earlier chapters are dated snapshots: the session of May 19, the as built of June 11 and the engine change of September. This is the system as of October 24, 2026, surveyed on the production server and database with reads only, on commit f41235c7 of **main**. It is a beta running on test data: what follows is architecture and configuration, not usage metrics. On October 26 we added the pruning of the Inngest history and the answer to "Inngest or a cron job in Postgres?"; since the survey, the code on main changed only in documentation.

**The system in one sentence:** a 3D office on Vercel where every user has their own workspace and their own team of agents, on top of a substrate in a Hetzner box that turns each request into a typed graph, runs it on its own engine whose only authority is Postgres, reaches models only through the Vercel AI Gateway, records every step and every call and, in 25 of its 27 templates, waits for a person to approve before a deliverable counts as done.

| | |
|---|---|
| **7** | Nova outcomes |
| **75** | Operations id@version |
| **27** | Templates |
| **12** | Evaluators |
| **18** | Inngest functions |

### A · The engine: Postgres is in charge

#### Engine v2 · sole authority

**What it is.** At launch, the compiler turns the template into a graph (`graphs` and `graph_nodes`) and starts a run in `runs`. Everything that happens to it lands in `run_events`, in order and without gaps (29 event types, 9 run states), and every ready node enters the `work_items` queue with a lease. A worker takes up to 5 items per batch and runs them in parallel, authorizes each node again against the roster before running it, and retries a failed node up to 3 attempts in total.

**What was left behind.** Engine v2 has been the only one running since September 5, 2026, and `seleccionarMotor` only returns v2 or fails closed. The v1 executor is out of the Inngest registry; its file is still in the tree, marked `@deprecated`, and a test fails if anyone registers it again.

`apps/api/src/orquestacion/ · runs · run_events · work_items · v1-inalcanzable.test.ts`

#### Inngest · durable intake and clock

**What it still does.** 18 registered functions: 7 that respond to events (the declared intent, the asynchronous compose, FORGE and the canary, among others) and 11 crons. The key one is `orquestacion-worker`, which wakes the engine's worker every minute. None of them holds the state of an engine run: if Inngest stops, what was already decided stays in Postgres, but the work stops moving (see the lesson below).

**What is not automated.** Changing a function's id, triggers or concurrency requires a manual resync with Inngest: neither CI nor the automatic deploy does it.

**What about a cron job in Postgres?** The answer is in block I.

`apps/api/src/inngest/functions/ · substrate-inngest + substrate-inngest-redis · 127.0.0.1:8288`

### B · Nova and the catalog

#### Nova · 7 outcomes

**What it decides.** One call to the model of the `compose` class resolves each request into one of seven typed outcomes: `converse`, `gate`, `match`, `match_custom`, `plan`, `cannot` and `invalid`. Compose never executes: it returns a proposal that the person launches. In production it is asynchronous (`COMPOSE_ASYNC=on`): it answers `202` with a `task_id` and the web polls the task until it finishes.

**FORGE.** It is no longer an outcome: it is a branch of `cannot`, switched on in production (`FORGE_ENABLED=true`), capped at 2 forges per workspace per hour, with human consent before spending and a second human gate before the capability is registered.

`apps/api/src/substrate/nova-compose.ts · compose-service.ts · apps/api/src/forge/`

#### The closed catalog · 75 · 27 · 12

**Operations.** 75 `id@version` pairs over 71 ids (four live in two versions) and 71 registered handlers. The ones without a handler are deliberate: the engine resolves the two human operations (`human_gate.approve` and `human_task`), and `slack.post` and `github.gist_create` only run if FORGE registered that capability.

**Templates and evaluators.** 27 exported templates, plus two derived tails (recovery and repair); 25 end in a human gate, and the two without one are `document-query-v1` and `pricing-watch-v1`. Nova has 10 curated workflows for `match`. The 12 evaluators run without a model: even the one declared `llm_judge` uses a deterministic implementation, and the judges that do use a model live outside `evaluator.run`.

`packages/substrate-spec/src/operations/catalog.ts · templates/index.ts · evaluators/catalog.ts`

### C · The research brief

#### The live graph · brief-synthesis-v13

**What runs.** Nova resolves a brief request with the `content-brief-full` workflow, and the graph version depends on research flags. In production the chain is switched on up to v13 and not v14, so the live graph is `brief-synthesis-v13`: 25 nodes, 43 edges, one evaluator and a gate with two phases. Every engine v2 run in the production database is on that graph.

**What it does.** It reads the audience; it researches YouTube and Reddit through ScrapeCreators, and search demand, the SERP and competitor metrics through DataForSEO; it synthesizes the research (`r4`), writes the brief (`s4`), evaluates it (`s5`), publishes it (`s6`) and leaves it at the gate (`s7`). The research budget in production is 1.50 USD per run.

`packages/substrate-spec/src/templates/brief-synthesis-v13.ts · handle-intent-declared.ts`

#### A gate in two phases · human review

**How it gets approved.** In phase 1 the person reviews the brief and can correct it inside the review; since migration 0126, the research synthesis too. Every version is kept in `revisiones_de_brief` and the original is never touched. In phase 2 they pick the angle. The deadline is 72 hours; if phase 2 expires, the system never picks for the person, and the only fallback that launch accepts is `fail`.

**If something breaks.** A run that dies before publishing has a recovery tail of 4 nodes (`s4` to `s7`); a report that was already published has a repair tail of 10 nodes. Both require the owner's delegated principal. The report downloads as a PDF from the «Imprimir / guardar PDF» button (PR #167, October 10).

`apps/api/src/inngest/gate-two-phase.ts · routes/revision-del-brief.ts · migrations 0125 and 0126`

### D · Multitenancy and security

#### One workspace per user · delegation

**What changed.** In June the beta served a single workspace; today every user has their own (`user_home_workspaces`, resolved with `get_user_home()`) and the API runs with `MULTI_TENANT=on`. On every request the web signs a delegation token (HMAC) with the `user_id` and the workspace, bound to the method, the route and the body, with a maximum life of 120 s. With `DELEGATION_ENFORCE=on` the API rejects anything without it and checks that the user is a member of the workspace.

**What it is not.** The 17 workspaces in production today belong, almost all of them, to test accounts that are reset often. This is isolation that is built and switched on, not traction.

`apps/api/src/routes/tenants.ts · middleware/delegation-membership.ts`

#### RLS in the database · 70 of 85 tables

**What it does.** Of the 85 tables in `public` in production, 70 have RLS enabled and 1 has it forced. The policy filters on `app.workspace_id` and fails loudly if it is missing; `withWorkspace` sets it on every transaction. The runtime uses the `substrate_app` role, without `BYPASSRLS`; lease reclaiming, maintenance, resets and the admin console have separate roles, each with its own URL.

**Before running anything.** The worker authorizes each node again against the current roster (`autorizarEjecucion`), and Presidio anonymizes documents before they reach the vectors; if Presidio is down, the original stays in quarantine.

`migrations 0099 to 0125 · apps/api/src/substrate/db.ts · autorizacion-de-operacion.ts`

### E · Models: everything through the gateway

Production reaches a model only through the Vercel AI Gateway, and not by configuration but by construction: outside development mode, the CLI, direct Anthropic API and Bedrock strategies throw an error; the service PATH has no Claude CLI; CI checks with static analysis that no production module invokes it, and `LLM_API_FALLBACK=true` prevents startup. The model is chosen by class:

| Class | Model | Used for |
|---|---|---|
| compose | `spacexai/grok-4.6` | Nova's compose, FORGE, single task analysis and some research operations |
| discovery | `anthropic/claude-opus-5` | Nova Discovery: proposes 2 or 3 agents during onboarding |
| cortesia | `anthropic/claude-haiku-4.5` | Kickoff, the question after a rejection and the Nova panel in the admin console |
| standard | `anthropic/claude-sonnet-4.6` | Chat, charter and most text operations; the `lite` and `bulk` classes use the same model |
| synthesis | `anthropic/claude-sonnet-4.6` | Research synthesis (`r4`) |
| keywords | `anthropic/claude-opus-4.8` | Keyword expansion (`r0`) |
| judge | `anthropic/claude-opus-4.8` | Judges: `document.query` faithfulness and the board panel |
| content_brief | `anthropic/claude-opus-5.5` | **Pinned in production.** Writing the brief (`s4`), with the `v15_dual` contract; the code's base profile is Sonnet 4.6 |
| voice | `anthropic/claude-opus-4.8-fast` | **Pinned in production.** The think step of the voice call; it is not a class, the `VOICE_THINK_MODEL` variable sets it |

Except for the two marked rows, these are the code defaults. The `model_routing_policy` table can change any class except `content_brief`, and its production rows were not surveyed. Jev, the router for node `s4`, runs in shadow: it decides and gets logged, but it does not change the model. Outside the LLMs, `document.query` reranks with Cohere `rerank-v4.0-pro` (active in production) and embeddings are local: `multilingual-e5-small`, 384 dimensions, inside the process.

### F · Observability and operations

#### Honeycomb and Phoenix · traces

**What they see.** OpenTelemetry exports to Honeycomb (service `agentsquadtraces`, environment `prod`), with one span per LLM attempt and per paid effect. Phoenix runs on the same server, in three containers, and receives metadata only, never prompts.

**What went away.** Langfuse was retired on August 22, 2026: its containers (web, worker, ClickHouse and MinIO) were removed. The code keeps a client that does nothing without a server.

`apps/api/src/observability/otel.ts · phoenix.ts · ~/substrate-infra/langfuse/ (configuration only)`

#### llm_calls · accounting

**What it stores.** One row per model call: model, tokens, cost and outcome, never the prompt. Charges from research providers go to `research_provider_attempts`, and the timeout shadow (`TIMEOUT_SHADOW=on`) measures each attempt against the gateway in `llm_attempt_shadow` without cutting anything.

**What it does not say yet.** The code has estimates per call and caps, but no typical cost per run has been measured: it has to come from these tables, not from a number someone wrote down.

`migrations 0113 (llm_calls) · 0082 (research_provider_attempts) · 0122 (llm_attempt_shadow)`

#### SLOs and alerts · every 10 minutes

**What it watches.** The `slo-alert.ts` cron compares the SLO snapshot with its targets, warns if a metrics source fails, checks whether FORGE is stuck and runs the canary of the asynchronous engine; a dead canary and a v1 executor that wakes up again are the two critical alerts. It notifies through Telegram, with a cooldown of 6 hours for warnings and 15 minutes for critical alerts.

**Known blind spots.** `learn-failures.ts` (08:00) groups failures from `step_executions`, a table that engine v2 does not write, and two checks in `slo-alert.ts` read journald, where the API does not write. They are documented as limitations, not as coverage.

`apps/api/scripts/slo-alert.ts · learn-failures.ts · docs/runbooks/alerting.md`

#### Backups · PITR

**What runs.** A daily `pg_dump` at 03:00 with a copy to Cloudflare R2; an incremental pgBackRest at 04:00 with continuous WAL to R2, which allows recovery to a point in time; and a restore drill every Sunday at 06:00. On October 24 both backups of the day were OK, and the last successful drill was on October 18 (99 tables).

**What is missing.** There is no hot replica: the RPO depends on the WAL (minutes) and on the daily dump.

`substrate-infra/scripts/restore-drill.sh · docs/runbooks/disaster-recovery.md`

### G · Built but switched off, and what was retired

- **The tool plane.** A permission broker with signed grants that last 5 minutes, bindings per workspace and adapters to run external tools (Slack, GitHub) on the server itself or on AWS AgentCore Gateway. In production `TOOL_PLANE_ADAPTER`, `AGENTCORE_GATEWAY_URL` and `TOOL_BROKER_GRANT_SECRET` are not defined: it exists, but no agent uses it. *(off)*
- **FORGE on Eve.** The second FORGE engine, on Vercel, is still in `experiments/forge-eve/`, unchanged since June 19 and outside the live path. The native FORGE is switched on. *(off)*
- **Jev and the timeout shadow.** Both observe and log without deciding: Jev does not change the model for `s4` and the shadow does not cut calls. The mode that would cut them is forbidden in production. *(in shadow)*
- **Apify and Tornado.** Their clients exist and their keys are configured in production, but no operation uses them. *(unused)*
- **Retired.** The v1 executor (it stopped running on September 5, 2026); Langfuse (August 22, 2026); Claude Code through the CLI as the production generator, now blocked by construction; GitHub as origin and CI, replaced by its own Forgejo server on August 24, 2026; and the paid embeddings with 3072 dimensions, replaced by local ones in June (ADR 0001). *(retired)*
- **The public demo in three acts.** The «Live demo» section of this page calls `demo.digitalhubassist.ai`, which does not resolve in DNS today: the demo does not work. *(broken)*

### H · Recent operational lessons

#### Inngest can get stuck with no visible errors · October 8 to 24

**What happened.** From October 8 to October 24, 2026 the Inngest queue stopped moving in stages: first the orchestration worker, then the canary and finally the rest of the crons. The container showed as healthy, the app showed as connected, and the Inngest logs only showed `publishing event`. The canary's critical alerts reached Telegram for days, but nobody acted on them until someone checked the canary by hand.

**How it is detected now.** The signal was there: the canary of the asynchronous engine, one of the two critical alerts of the SLO cron. What is new is the procedure to confirm it without relying on the Inngest logs, which do not record executions: count the `POST /api/inngest` calls in the API log at two moments, and look at each function's queue in Redis (`ZCARD`, and the score of the first item, which is its scheduled time: if it is far behind and does not change between two reads, the queue is not moving).

**How it is fixed.** Before restarting, check the cron schedules: after a stall, overdue entries remain, and on restart each one catches up minute by minute. On October 24 that created 800 worker runs in 11 minutes, and the history grew 17 MB per minute with the disk at 99%. The procedure now is to stop Inngest, back up Redis, remove those overdue schedules and the pending runs that have no work, and only then start it. Redis is never wiped entirely, because the queue lives there.

**What we did on October 25.** The old history, 19.7 GB, was archived to Cloudflare R2, with its sha256 checked after downloading and decompressing it, and Inngest started with a new history that keeps only the function configuration. Rotation is now a command that is run by hand, with a short stop: it builds the new database and checks that the crons are still alive; if that check fails, it tries to go back to the original, and an interruption halfway through is repaired by hand with the same command. An hourly alert sends a Telegram message when the history passes 2 GB, about 6 days at the current rate.

**What is still open.** The likely cause of the stall, a SQLite history with no limit, is contained but not solved: it grows about 330 MB per day, and 85% of it comes from four crons that run every minute and almost never have work. The plan to harden Inngest is in block I.

`docs/runbooks/alerting.md, section «Cuando el canario falla»`

#### Monitoring moves with the engine

Three guards were still looking at the v1 world: `learn-failures.ts` reads `step_executions` and also fails on its query; the live run guard of the automatic deploy counts running `traces`, which engine v2 does not write; and two checks in `slo-alert.ts` search journald. None of them fails loudly: they simply see nothing. They are now listed as known limitations in the TRD.

#### The code default is not production

With the flags off, the code builds the brief with v3 and leaves FORGE off; production runs v13 with FORGE on. That is why the October documentation labels every statement as active in production, code default or unused, and this chapter spells it out wherever the code and production differ.

#### What is manual is still manual

Migrations are applied by hand in production before merging the PR that uses them, and there is no version table; CI applies all of them, in order, on a test database. The Inngest resync is not automated either. The API deploy is: a cron every 3 minutes that only deploys when CI is green.

### I · Inngest or a cron job in Postgres?

It is the question we get most often from people who read this document: "Inngest doesn't work well, you could replace it with a cron job in Postgres." We answered it with the code on **main** and with production reads from October 26, 2026, and put the answer through an adversarial review by two other models (Grok 4.6 and GPT-6 Astra).

#### What Inngest does today · 18 functions

**Mostly, it is a clock.** 11 functions have a cron (9 of them also accept an event) and 7 respond only to events; 12 of the 18 have a single step. The v2 engine worker runs its whole batch inside one step and with no retries: the durability of a run already lives in Postgres, with a lease, `FOR UPDATE SKIP LOCKED`, a heartbeat and a watchdog.

**Where it really is a durable engine.** `handle-intent-declared` memoizes its steps, so a retry does not repeat the ones that already finished (starting the run is also idempotent in Postgres); the shadow board reuses the model stages that Inngest already confirmed, although if the response is lost after the call, the stage can run again; and FORGE has two waits for human approval of up to 72 hours each, with `step.waitForEvent`.

`apps/api/src/inngest/functions/ · worker-orquestacion.ts · forge-capability.ts`

#### Why not a cron job in Postgres · pg_cron

**It schedules SQL, not the work.** `pg_cron` runs SQL inside the database and records those runs. The real work is TypeScript: model calls through the gateway, tools and video. Running it would need a TypeScript consumer that reads a queue, or calling the API over HTTP with `pg_net`, which this image does not ship. Either way, tracking, retries and concurrency for the work would still have to be built.

**And it costs more than it looks.** `pg_cron` 1.6 ships with the image but is not installed: installing it means changing `shared_preload_libraries` and restarting the production Postgres. Even then it would not cover the 7 event functions, the concurrency limits or the FORGE waits.

`substrate-postgres (timescaledb-ha pg16) · read on 2026-10-26`

#### The serious alternative · assessed, not now

**A job runner on Postgres.** Really replacing Inngest would mean a TypeScript job runner on Postgres, such as graphile-worker or pg-boss: a queue with retries, `LISTEN/NOTIFY` to wake up instantly, and enqueuing in the same transaction as the state change. It fits the v2 engine.

**What would have to be ported.** Step memoization, the two FORGE waits of 72 hours, event idempotency and concurrency per function. It is a medium or large project, and in the last 30 days there was a single v2 run.

`orquestacion/despachador.ts · substrate/db.ts (LISTEN already used for compose)`

#### What the critics got right · the hosting

**It caused October's worst incident.** Inngest was stuck for 16 days with no errors in its logs (block H). Part of the pain comes from how we host it (a SQLite history with no retention, an unpinned image, crons that run every minute and are almost always empty, and a race with Redis when Docker restarts) and part from the integration (the HTTP connection with the API, a worker that nobody wakes up, and the manual resync). The SQLite history is the likely cause of the stall, still unconfirmed.

**And a risk still to check.** The API server closes idle connections after 255 seconds, and the log shows worker batches of up to 898 seconds inside a single Inngest call. We still have to check whether Inngest received those responses.

`apps/api/src/index.ts (idleTimeout: 255) · apps/api/.runtime/stdout.log`

#### The decision · 2026-10-26

**We keep Inngest and harden it.** Pin the image version, start Inngest only after Redis finishes loading, wake the worker with an event when work is enqueued, move the four crons that almost never have work from every minute to every 5 minutes, and check the risk of long batches. Moving the history out of SQLite comes later and is reassessed with data: how fast it grows, whether write errors appear and whether rotation is still viable; sooner if the stall happens again. Every change is tested first on a rehearsal Inngest, and nothing has changed in production yet.

**When the migration question reopens.** If the stall comes back after the hardening, if a batch is cut off halfway with work already paid for, or when FORGE is rebuilt on the v2 engine gates. A date is not a criterion.

`adversarial review with Grok 4.6 and GPT-6 Astra · 2026-10-26`

### May → today · what the cover said versus 2026-10-24

| Topic | May (cover) | Today |
|---|---|---|
| Primitives | 10 | 11: `audience` was added |
| Operations | 16 | 75 `id@version` pairs over 71 ids |
| Templates | 4 | 27, plus two derived tails for recovery and repair |
| Workflows | 4 in production | 10 curated workflows for Nova's `match` |
| Engine | Inngest walks the DAG | Engine v2 on Postgres; Inngest remains as durable intake, crons and the worker's heartbeat (18 functions) |
| Model | Sonnet 4.5 | Only through the Vercel AI Gateway, one model per class; Opus 5.5 writes the brief |
| Data | Postgres + pgvectorscale | Postgres 16 with pgvector 0.8.2 and vectorscale 0.9.0; 85 tables, 70 with RLS enabled; local embeddings with 384 dimensions |
| Cost | $0.06 for the smoke | No typical cost per run measured yet; a research cap of 1.50 USD per run, and every call is recorded in `llm_calls` |
| Tests | 28/28 passing | 880 test files in `apps/api`, 284 in the web app and 92 E2E specs; both deploys require green CI on Forgejo |

> The May thesis still stands: **closed catalog, nothing unsigned, never invents.** What changed is where the truth of a run lives: no longer in the history of an Inngest function, but in an event log in Postgres that can be folded, audited and rebuilt after a crash.

Sources: the production survey of October 24, 2026 (reads only) and, in the repository, `docs/ARCHITECTURE.md`, `docs/CONCEPTS.md`, `docs/reverse-engineering/02-trd.md` and `docs/runbooks/alerting.md`, updated to that date. Any figure in this chapter that goes stale is a bug in the document.

*Next turn · agenda as of 2026-10-26*

## What comes *next*

The agenda of August 19 was met in the big things: the v2 engine has been the only one since September 5, and every user has their own workspace with RLS. What comes next is what chapter 15 found still open, ordered by risk.

**Priority**

A. **Harden Inngest.** Pin the image version, start after Redis, wake the worker with an event when work is enqueued, move the crons that run every minute to every 5 minutes, and check whether long batches outlast the connection. Moving the history out of SQLite is reassessed later, with data. Plan reviewed by two models, no changes in production.
B. **Move monitoring to the v2 engine.** `learn-failures.ts`, the guard for live runs in the automatic deploy and two checks in `slo-alert.ts` read tables or logs that the v2 engine does not write: today they see nothing.
C. **Measure the typical cost per run** from `llm_calls`, instead of estimating it.

**Open according to the survey**

01. **Inngest resync after every deploy**, with a check that fails if the registered functions do not match the code.
02. **Migrations with a version record.** Today they are applied by hand in production and the only source of truth is the schema.
03. **A hot replica of Postgres.** Today recovery depends on continuous WAL and the daily dump.
04. **The public demo.** Bring it back up or remove it from this page: its domain does not resolve.
05. From the August 19 agenda, not reviewed in this survey: deleting `execute-plan.ts` and cleaning up the legacy schema, the offline evaluation sustained over time, measuring the rate of gates without a decision again, and the `email-triage` template, which does not exist in the catalog.

**Already closed**

✓. **Inngest history pruned**: 19.7 GB archived to R2, manual rotation and a 2 GB alert.
✓. **One workspace per user and RLS** on 70 of the 85 tables, with a runtime role without `BYPASSRLS`.
✓. **v2 is the only engine** since September 5; the v1 executor is out of the Inngest registry.
✓. **Models only through the Vercel AI Gateway**, by construction and checked in CI.
✓. **Own origin and CI on Forgejo** since August 24; both deploys require green CI.
✓. **Durable gate in two phases**, with handoff, ACK, required role, deadline and sweep.

*The substrate, in your hands · guided demo*

## Give your team a task and *watch* it work

> **Out of service.** The public demo calls `demo.digitalhubassist.ai`, which does not resolve today: the buttons on the web page will not respond until it is brought back up.

Three acts against the real engine. You give it a goal in natural language; the team plans it and executes it; Quality control reviews it *before* delivering it; you approve or ask for changes. Then you ask for something your team does not yet know how to do, and it builds it.
