validatePlanAgainstCatalog), a graph with no cycles via Kahn, a cap of 10 steps. A human did not draw this graph; the system derived it and proved it.A symbolic substrate
for agents that compose
This document is a chronicle: each chapter is a dated snapshot, starting with the session of May 19, 2026 that this cover describes. Today's system is in chapter 15.
The "v3" in the address is the third edition of this document, not the engine version. The engine running in production is v2, the only one since September 5, 2026.
Why Inngest and not a cron job in Postgres? The answer, with evidence from the code and from production.
Figures from the session of May 19, 2026. The current state is in chapter 15.
Want it as text? Download the full document as Markdown (.md · ~72 KB · without the SVG diagrams)
What is a substrate?
A substrate is the layer beneath agents and workflows where everything they produce ends up typed and queryable. It is not a database. It is not a knowledge base with embeddings. It is a graph where every decision, every output, every human judgment, and every inference an agent asserts is deposited as a node with explicit relationships to the others.
The difference from a conversational assistant is direct: the assistant is amnesiac, the substrate is consistent with its own past. Every new answer can query everything that was decided before and reason against it. The approval you gave yesterday is context available to the agent that starts tomorrow.
(subject, predicate, object), not clouds of numbers. Every assertion is defensible because it has structure and provenance.↓ (nothing is recorded)
Workflow B starts with no context
→ repeats questions, contradicts earlier decisions
- Every conversation starts from zero, even between agents on the same team
- The user acts as the system's memory (you paste context into every prompt)
- Decisions evaporate: nobody knows who approved what, or why
- Assertions cannot be audited: the model said X, nobody knows where it got it
↓ (the graph grows)
Workflow B reads A's Claims before starting
→ reasons against earlier decisions, not against zero
- Every human decision materializes as a Claim that future workflows can query
- The team of agents accumulates structured memory: not just logs, typed knowledge
- Native traceability: any assertion has provenance (which Trace produced it, which human validated it)
- Network effects: every extra workflow adds context for all the ones that follow
It started with a talk
at Stanford
8090
Tools wear out; substrates compound. What Chamath called a control plane was not a product but a layer beneath the agents where every decision, claim, intent, and artifact is traced and available for future inferences. That is what makes a team of agents improve with use.
The point was simple and hard at the same time: if I build the consumer Agent Squad without a substrate, I repeat the pattern of every AI SaaS: users produce pixels but the system does not learn. If I build the substrate first, every workflow I mount on top of it afterward compounds value in the graph. Consumer Agent Squad becomes the first ontology overlay, not the final product.
And then came the strategic piece. Chamath was connecting the control plane to network effects: the graph of typed Claims is what creates the compounding effect across companies, not the app mounted on top.
We decided not to build a Photoshop. The product is the substrate. The app Miles is finishing is the first public demonstration of what that substrate makes possible. The underlying idea: { Intent, Plan, Trace, Claim, Artifact } are the primitives that any agentic workflow can compile into an executable DAG, and any human decision becomes typed context for the next one.
Seven strategic decisions
made fast
Ten primitives and two hard rules
knowledge_access.requires_vectorTwelve tech decisions
made in an hour
| Layer | Tech | Why |
|---|---|---|
| Spec | JSON Schema + Zod + TypeScript | Open spec, no proprietary DSL. Anyone with a valid JSON Schema can produce outputs without touching the engine. |
| Engine | v2 event sourced + Inngest OSS | The compiler produces graph_nodes; a pure reducer + apply-event project runs/run_events; a worker consumes work_items. Inngest keeps durable events and crons. execute-plan is the v1 executor, being deprecated. |
| Substrate DB | Postgres 16 + pgvector + pgvectorscale | DiskANN for 3072-dim embeddings (HNSW tops out at 2000). Dedicated on Hetzner, NOT on InsForge. |
| Workspace DB | InsForge | Still used by the consumer app (auth, user data, outputs feed). Separate from the substrate by design. |
| LLM | Max CLI (claude-cli, Max plan; no Anthropic API) | Sonnet 4.5 and Opus for composers and scorers; Grok for synthesis. The Vercel AI SDK + @ai-sdk/anthropic path stayed as a dormant gateway (Phase 1). |
| Embedding | OpenAI text-embedding-3-large (3072d) | Best precision/recall for semantic retrieval. Backup to R2 from day one in case we migrate the vector DB. |
| Reranker | Cohere Rerank v3.5 | Relevance layer over the top K. Can be turned on as the graph grows. Not needed yet. |
| Evaluators | Built in house in packages/substrate-spec | No LangSmith. Evaluators are tested code that lives with the spec, not an external service. |
| Observability | Langfuse OSS v3, self hosted | Web + worker + ClickHouse + MinIO + Redis + Postgres. Tracing by the substrate's trace_id. |
| API runtime | Hono on Bun 1.3 | Zero cold start, minimal footprint. A single Hono app exposes /api/intents, /api/approvals, /api/health. |
| API contract | internal tRPC · external REST + OpenAPI 3.1 | Shared types internally. OpenAPI externally so any client can implement it. |
| Compute | Apps on Vercel · Engine + DB + Obs on Hetzner | Hetzner 4CPU/8GB already paid for. The stateful engine lives where we have disk; stateless apps stay on Vercel. |
One Hetzner box,
every service running
The backup runs nightly at 3 AM via cron on ~/substrate-infra/scripts/backup-substrate-db.sh. It writes a dump to ~/backups/substrate/ and, if the rclone remote r2-substrate: exists, also pushes to Cloudflare R2. The substrate's SQL migrations live in agent-squad-app/db/substrate/migrations/0001_init.sql: 15 tables, DiskANN indexes over the vectors of Claims and Artifacts, date partitioning on traces and step_executions.
Think of it this way: imagine the brief you approved yesterday is hanging on a wall. Next to it, sticky notes keep appearing, written automatically by the system. Each of those sticky notes is a Claim.
| Part | What it answers | Examples |
|---|---|---|
| subject | What the sticky note is attached to | a brief · your voice · a project · a decision · a prospect |
| predicate | What it says about that thing | approvedBy · hasKind · voiceTone · fitScore · producedByOp |
| object | What value it carries | "human:roberto" · "doc" · "direct, no detours" · 0.75 |
| provenance | Where it came from (native audit) | which Trace emitted it · which Step · which human confirmed it |
(brief 3aea, approvedBy, human:roberto) + provenanceclaim.recall_decisions over the graph and hands the model: "1 day ago, on a similar brief, Roberto approved the Hook 2 variant with the comment 'beats the other 3'". The LLM does not guess or remember: it inherits that decision as input. After 100 workflows you don't have 100 logs or 100 opaque vectors: you have a graph of 500 to 1000 typed assertions about your voice, your criteria, your past decisions. And any future workflow can query it before acting. That is the compounding effect.
Four well known alternatives
and where they break for this case
A library of capabilities
that any Plan can draw from
- research.synthesize@1.0.0
- research.keyword_expand@1.0.0
- research.reddit_search@1.0.0
- research.youtube_comments@1.0.0
- research.rank_opportunities@1.0.0
- research.audience_intelligence@1.0.0
- document.ingest@1.0.0
- document.chunk@1.0.0
- document.extract@1.0.0
- document.query@1.0.0
- media.ingest@1.0.0
- media.transcribe@1.0.0
- url.fetch_transcript@1.0.0
- video.script_draft@1.0.0
- broll.generate@1.0.0
- heygen.generate@1.0.0
- voice.tts@1.0.0
- junta.panel@1.0.0
- junta.compose_verdict@1.0.0
- junta.load_report@1.0.0
- text.compose_narrative@2.0.0
- text.compose_brief@1.0.0
- trace.query@1.0.0
- artifact.publish@1.0.0
- claim.recall_decisions@1.0.0
- claim.recall_voice@1.0.0
- evaluator.run@1.0.0
- prospect.search@1.0.0
- pricing.observe@1.0.0
- metrics.digest@1.0.0
- audience.load@1.0.0
A representative sample. The full catalog is 73 Operations registered in the engine (inngest/operations), with research as the densest domain.
Four production DAGs
each with its own agent
Git (commit history), Excel (references between cells), Airflow and Temporal (data pipelines). The engine topo-sorts the DAG so each Step starts only when all its inputs have finished, which lets several branches run in parallel.The durable gate
where the person is the final authority
step.waitForEvent: ACTIVAR_GATE creates a durable flow in artifact_approval_flows. Review, approval/choice, executor acknowledgment, and closing are persisted states; deadline_at + the sweep apply the fallback with CAS, and required_role defines who can decide.Critical bug learned: the match in step.waitForEvent must use async.data.X, not event.data.X. event references the original trigger (plan.compiled), so using it there evaluates at suspension time, not at the future event, and matches against null. async.data references the event being waited for. Without this, every gate hangs indefinitely.
What this unlocks: the human can approve, reject, or add a comment, and the decision is stored as a typed Claim that future workflows can query. The next time we build a brief on the same topic, claim.recall_decisions already has "this angle won" as context. This is the compounding effect of the substrate: it is not retrieval, it is structured memory construction.
Four templates activated
with real Sonnet 4.5
The nontrivial part: Sonnet reasoned about the workspace's own backlog. The standup-digest narrative literally mentioned how many earlier digests were left queued and awaiting_human, because trace.query + artifact.list_recent + claim.recall fed it that data. The retrieval → composer chain is validated end to end with real substrate data.
We approved the brief
and five Claims were born
"Substrate vs RAG"
This is the moment where the substrate stops being theory and starts behaving like capital. The next time we build a brief on the same topic, claim.recall_decisions will retrieve "Hook 2 won over the other 3" as composer context. Sonnet does not need to remember it: the graph hands it over. That is what sets a substrate apart from a RAG: it does not search for similar documents, it serves typed assertions with provenance.
After a single session, Roberto's workspace has four production PlanTemplates, seventeen atomic Operations, sixty or so Claims accumulated as compounding context, and a real operating cost of roughly two cents per executed workflow. That is not the important part. The important part is that every one of those Claims will still be here tomorrow, and a year from now, feeding the next workflows we build.
The substrate absorbs
a community standard
A plain Markdown file (AUDIENCE.md) versionable in git with twelve canonical sections: name, summary, primary audiences, jobs-to-be-done, pains, motivations, decision criteria, language and tone, anti-goals, evidence vs assumptions, open questions. Optional YAML frontmatter with status, owners, last_reviewed. Designed so that any LLM agent reads it as base context before operating on the audience.
Zero new primitives. Four clean alignments.
Conceptually equivalent to any other published Artifact, except that its lifecycle is "always present" instead of "produced by an Op". It lives on the filesystem, not in R2.
Defines which predicates are valid for talking about the audience (hasJob, hasPain, hasDecisionCriterion). Other workspaces can have different overlays.
This is the perfect match. Evidence with a verifiable source = Claim with high confidence + populated source_refs. Assumption = low confidence + empty source_refs. The standard and the substrate describe exactly the same distinction.
Meets Rule A: declarative, versionable, durable. Identical philosophy to the workspace manifest.
One new step. Zero refactors to the rest of the DAG.
The first AUDIENCE.md derived from Roberto's brand blueprint.
--- audience_id: ai4managers status: stable owners: [roberto@aguirre] last_reviewed: 2026-05-20 source_doc: basic-memory:/sistema/reference_ai4m_brand_blueprint.md --- # Audience: AI4Managers ## Summary Spanish-speaking middle-to-senior managers (40-55) who built consolidated careers before the AI shift and now sense the rules of their profession changed... ## Jobs To Be Done - Understand what AI changes in MY specific role this quarter - Operate AI tools and agents without becoming an engineer - Preserve professional standing against AI-fluent juniors ## Language And Tone **Core terms**: operar, desplegar, agentes en producción, manager, battle-tested **Prohibited terms**: "estamos en la era de…", "imaginá si…", "lab", emojis, jerga ingenieril sin traducción (LLM, RAG, embeddings, fine-tuning) ## Anti-Goals - Beginner-tutorial content - Engineering deep-dives that require coding fluency - English-only or US-only case studies ## Evidence - Roberto's professional transition (Manager Senior → Head of AI Ops Engineering) - Corporate training engagements (only currently validated monetization) - LinkedIn outreach with ~207 unique authors, 33 batches ## Assumptions - LATAM and Spain share enough vocabulary to be a single audience - Status preservation > upside ambition (untested at scale) ... 4 more sections: Motivations, Decision Criteria, Open Questions, Content Pillars
The prompt now has a precedence rule and a pillar guard.
- If an AUDIENCE DEFINITION block is provided in the user message, it is the
authoritative source of audience truth. The short "Target audience" label is
only a hint.
- audience_profile MUST reflect the canonical document: pains, motivations,
decision criteria, jobs-to-be-done.
- angle, hook_patterns and key_message MUST respect Language guidance:
use core terms, avoid prohibited terms verbatim.
- If the topic does not fit any allowed content pillar declared in the document,
set ready_for_production=false and put a one-line reason in key_message
starting with "PILLAR_MISMATCH:"
The pillar guard is the interesting piece. If someone dispatches an Intent to write, say, a basic prompt engineering tutorial (which fits none of the four pillars in the brand blueprint), Sofia does not produce the brief: she flags it PILLAR_MISMATCH with a reason. The human gate receives it and Roberto decides whether to widen the pillars or discard the piece.
Workspaces without AUDIENCE.md keep working exactly as the day before. The handler returns { present: false, doc: null } and Sofia falls back to the target_audience string as she always did. The previous smoke tests (standup, lead, brief without audience, video) pass untouched. The extension is turned on per workspace, not a global behavior change.
120 lines inserted. 6 files modified. 5 new files. 6 more tests.
The architectural reading. This is the first time an external community standard maps cleanly onto the substrate's primitives without forcing. That confirms a hypothesis we had been assuming: the symbolic model has enough expressive power to absorb community conventions without rewriting them. The next candidate extension is llms.txt (Jeremy Howard's standard for AI crawlers to consume docs). If it maps to Artifact + WorkspaceManifest again, the "absorb don't reinvent" pattern becomes doctrine.
The operational reading. The next produce_artifact / content-brief Intent dispatched against the AI4Managers workspace will generate briefs aligned with the brand blueprint without anyone having to paste the document into the prompt each time. The thesis, the pillars, the prohibited words, the anti-goals: everything is in the graph, and it is applied automatically.
The architecture today,
piece by piece
The system in one sentence: a stateless app on Vercel declares intents to an engine on a Hetzner box; the engine compiles them into typed plans, executes them durably with an LLM whose cost is accounted for step by step, forces a human approval at the end of every plan, and turns every human decision into structured memory you can query with SQL.
From the outside, a running plan looks like a graph of steps with one node that queries a vector database. Hence the easy criticism: "this is an n8n template with a vector node, and that's it." It confuses the artifact (a DAG at runtime) with the process that produced it. n8n is a flow that a human draws by hand. Here the flow is not drawn: it is composed and validated. Three differences of category:
fallback:'fail': the model has no path to skip it. It is neither "the LLM does everything" nor "everything hardcoded."The sophistication is not where critics point (that a DAG exists, that a vector exists) but where they do not look: validated composition, the catalog↔people inversion, and the structural gate. Four real architects (Cherny, Embiricos, Chase, Majors) put this as built system under review: minutes of the Master-Arq session →
app_state · user_access (gate) · magic_links (one time)proxy_buffering off for SSE · Inngest is NOT publicidleTimeout: 120step.waitForEvent ≤24h · survives restartstext_delta reaches the user~/backups/substrate/ (+optional R2) · standup digest 07:30 with CLI preflightThree compute domains, each where it fits best: the stateless part on Vercel (scales by itself, stores nothing), the stateful part on a box with a disk (Postgres, Inngest, Langfuse: 3 Docker Compose stacks plus one systemd unit), and identity in a separate BaaS, decoupled from the substrate by design. Total observed memory on the engine side: ~1.7 GB of 15 available.
What it is. A full stack app with runes (compiled reactivity) served serverless. 13 product pages (welcome, onboarding, office, briefing, library, activity, outputs, squads, hire, squad-proposal, discover, deep-dive, share; the public demo, the OAuth callback, and the legal pages stay apart), each with its own +page.server.ts that loads real data from the engine and degrades to labeled sample content if the engine does not respond.
What it is for. It is the face of the product: everything the founder sees and touches. Stateless on purpose: it stores nothing, so Vercel can create and destroy instances without losing state.
What it is. A voxel scene rendered in the browser: 16 agents in 3 squad zones, 12 desks, a layout computed by a pure function (tested with vitest, without touching WebGL).
What it is for. It turns "you have agents working" into something you can see. The office recap ("N shipped · M await review") comes from an engine endpoint that aggregates real traces from the last 24h.
What it is. The chat with each agent. A custom SSE parser (~60 lines, pure, tested) paints each fragment of the reply into a transient bubble; when the drawer opens with an empty session, it hydrates the thread persisted in the engine.
What it is for. Talking to an agent with no process waiting: the agent's "continuity" lives in its memory (briefing + jobs + thread), which the engine assembles at the moment of the message. If the stream is cut before the first fragment, it degrades to the classic path with a single retry.
What it is. The "ask Nova" UI: a natural language textarea and a modal that shows the outcome: a match with an existing SuperSkill, a newly composed plan (with estimated cost and human steps), or an honest "I can't." From there the plan is saved under its own name.
What it is for. The user never browses a technical catalog: they ask. And what Nova composes can be promoted to a relaunchable SuperSkill from the library.
What it is. A backend as a service that provides auth.users with a passthrough JSONB profile, plus two tables of its own: user_access (a boolean gate for the private beta, fail closed) and magic_links (CSPRNG tokens, single use, 15 min, atomic consumption with UPDATE … WHERE consumed_at IS NULL).
What it is for. All of identity, separated from the substrate by design. The state of the user's app (squad, onboarding, installed SuperSkills) lives in profile.app_state: the profile IS the consumer's database.
What it is. The SvelteKit hook that runs on EVERY server request: it reads the httpOnly cookie, renews the access token with the refresh token if it expired (~15 min), and queries user_access with the service key.
What it is for. Truly fail closed: an error reading the gate means unauthorized. The bypass for E2E tests exists but is doubly gated (dev && CI): in a production build that branch is dead code.
What it is. Login, Google OAuth, and magic links are exchanged on the server (client_type=server), never in the browser. The reason: the browser SDK performs the PKCE exchange in its constructor and never exposes the refresh token, and without it sessions died after 15 minutes.
What it is for. 30 day sessions with silent renewal, and tokens that live only in an httpOnly/secure cookie.
What it is. The only endpoint that writes app_state: it does a read merge write of the full state, resanitizes fields with an explicit pick (junk does not persist), and rejects bodies >64KB. On the client side, a write queue serializes the POSTs (two writes in flight would clobber each other) with optimistic updates and a compare and swap revert.
What it is for. Multiple devices without a state server: you create a squad in one browser and see it in another. localStorage remains as a write through cache, with precedence profile > cache > derivation.
What it is. TLS with a Cloudflare origin certificate and a single nginx site that proxies 443 → 127.0.0.1:4000, exposing only /health, /api/workspaces/*, /api/approvals, and /api/intents. The Inngest endpoint does not exist for the internet. For chat SSE: proxy_buffering off and a 120s read timeout.
What it is for. So the engine has exactly the surface the app needs and not one path more. Port 4000 is closed to the public by firewall; only loopback and the Docker bridge reach it.
What it is. A server to server bearer token compared in constant time (anti timing attack), and an in memory sliding window rate limiter: chat 10/min, compose 4/min, intents 10/min, mounted after the bearer (a 401 does not consume quota), answering 429 with Retry-After.
What it is for. The LLM is the expensive resource; the limits protect the session from an accidental loop or an eager script. The browser never sees this token: only the server side app uses it.
What it is. A single HTTP app with 10 route files: POST /api/intents, POST /api/approvals, and under /api/workspaces/:id/ outputs, activity, brief (GET/PUT), manifest, chat (+history), compose (+launch/discard), superskills (+launch). It runs as a systemd unit with restart on failure and logs to a file.
What it is for. It is the only entry point to the substrate. A detail with a scar: idleTimeout: 120 in the server export, because Bun's default (10s) killed the SSE when the model reasoned silently for more than 10s. The live manual caught it, not a unit test.
What it is. Seventy three typed, versioned refs spread across 33 domains (trace.query, claim.recall_decisions, text.compose_brief, video.compose…); they are 70 ids, because three live in two versions at once. The largest domain is research, with 17. All have a runtime handler except two, human_gate.approve and human_task, which lack one on purpose: the executor itself runs them by suspending the workflow.
What it is for. The operation is the unit with a contract (typed inputs → outputs + cost) and the seam where any volatile dependency would be quarantined: an agent framework fits inside an operation; the reverse does not.
What it is. A declared Intent is compiled into a Plan: a topologically sorted DAG of steps where each step references a catalog operation. The plans Nova composes go through a 4 layer server side validation (every operation exists in the catalog, the graph has no cycles via Kahn, a cap of 10 steps, and the final approval is added by the system); every launch, whether factory, composed, or saved, revalidates against the catalog and structurally requires the terminal human gate. The evaluators emit verdicts per step.
What it is for. So that "the model decided" is never the explanation for an execution: what runs is a typed artifact the server validated, with an estimated cost computed server side (the model cannot lie about it).
What it is. A single query to the model decides: MATCH (the request corresponds to an existing SuperSkill, validated server side against the real list, so the model cannot invent ids), COMPOSE (builds a new plan using ONLY catalog operations), or CANNOT (says so honestly and the request is recorded as demand).
What it is for. It is the only place in the system where the LLM "designs," and that is why it is the most closely watched: what it proposes goes through its 4 layer validation and the same structural gate as everything else. Every request lands in plan_drafts as telemetry: the "couldn't do it" entries prioritize what to build.
What it is. A composed plan that the user liked is promoted under its own name into the workspace catalog: it is revalidated against the catalog BEFORE the insert, and a partial unique index in the DB makes a double promote impossible (exactly one wins, the other gets a 409). Launching uses a helper shared with the factory plans: assertHumanGatePresent structurally verifies ≥1 terminal human gate with fallback 'fail'.
What it is for. So that what the user creates by asking has the same guarantees as what the team built. The relaunch form is derived mechanically from the variables the plan itself declares.
What it is. A custom adapter with two backends: Claude Code headless via CLI (Max subscription session, marginal cost $0) and the official API as a fallback. For streaming it parses NDJSON line by line and emits ONLY text_delta: the model's internal reasoning never reaches the user; the final result line wins over what was accumulated.
What it is for. Separating architecture from stage: the beta runs at zero cost, and since every step records its notional cost in dollars, the unit economics of switching to the API are known TODAY ($0.013-0.022 per digest). Turning on the API is configuration, not rearchitecture.
What it is. The self hosted durable runtime for events, crons, and operational retries. handle-intent-declared selects v2 and starts a run over the compiled graph; worker-orquestacion drives the worker that consumes work_items; the sweeps recover expired leases and gates; materialize-manifest keeps its cron. execute-plan remains only as the v1 executor, being deprecated, and as an anti zombie canary.
What it is for. Inngest makes delivery and housekeeping durable, but it is no longer the authority over DAG progress. That truth lives in Postgres: runs + run_events, folded by the reducer, and the transactional queue the v2 worker processes. A deploy can interrupt a tick; it cannot erase what the run has already decided.
What it is. A dedicated instance (Docker, :5433) with 77 logical tables and 32 monthly partitions. The two highest volume ones, traces and step_executions, are partitioned by month. The embeddings (3072d, text-embedding-3-large) are a column in claims and artifacts with a StreamingDiskANN index: HNSW tops out at 2000 dimensions, which is why pgvectorscale.
What it is for. Semantic memory and typed facts in the SAME store: a JOIN between vector similarity and structured claims, with no dual write against an external vector DB. Consistency is free because there is a single database.
What it is. The chain intents → plans (steps, edges) → traces → step_executions → artifacts / claims, plus lineage_edges with a recursive view (max depth 12, cycle guard). A hard rule: every claim and artifact MUST have provenance {trace_id, step_id}.
What it is for. Native audit: for any statement the system makes you can answer "where did this come from?" by walking the graph: which trace emitted it, which step, which human confirmed it.
What it is. A navigable snapshot per workspace (squad, active intents, recent artifacts, decisions, navigation index) materialized every night and on demand by event. Since migration 0006, the manifest JSONB column is the source of truth.
What it is for. So an agent starts out knowing where it stands without listing half the database: it reads ONE record. It is a derived cache, and can be rebuilt entirely from the graph.
What it is. A self hosted stack of 6 services (web, worker, ClickHouse, MinIO, Redis, its own Postgres) that traces every LLM call with latency, tokens, and cost, correlated by the substrate's trace_id. With a scar of its own: ClickHouse capped at 2 CPUs after a runaway that pinned 7.7 cores for 19 hours.
What it is for. Answering "what exactly did the model do in this step and what did it cost?" with one click. The only Redis in the system lives inside this stack, and it is not in the app's path.
What it is. Two crons: a nightly backup (03:00 UTC, gzipped pg_dump to ~/backups/substrate/, 30 day retention, optional push to R2) and a daily standup digest (07:30 UTC, which declares a real Intent through the public API with a bearer, with a preflight that checks the CLI's health before firing).
What it is for. The backup is the answer to the catastrophic failure domain (RPO ≤24h). The digest is the product's heartbeat: the system uses itself every day. The dumps grow with the system: 680K in May, 1.8M today.
What it is. Three rings: 13,333 unit and integration tests (4,102 from the app, 9,058 from the engine against real Postgres, 173 from the spec), 443 E2E/visual tests with Playwright spread across 81 files (full journeys + screenshot regression), and the outer ring: a 34 step manual that a real browser reexecutes against production before every demo, with an ephemeral user and surgical cleanup of the workspace.
What it is for. The outer ring catches what the inner ones cannot see: the idleTimeout bug (SSE dead after 10s of silence) passed every unit and E2E test, and the manual found it. If a step breaks in production, its seal stops saying VERIFIED.
The static architecture is half the story. These are the three paths that teach the most, step by step:
| Table | What it stores | The why |
|---|---|---|
| intents | What who wants to achieve, in which workspace | The audited starting point of EVERY execution; supports a tree (parent_intent_id) |
| plans + steps + plan_edges | The executable DAG compiled from the Intent | Versioned per intent; each step carries operation_ref, retry policy, timeout, and an optional human_gate |
| operations | The catalog: metadata for each operation, versioned | Deprecate without deleting; the plan references id@version |
| traces | Each concrete execution of a plan (state, actual cost, verdict) | Partitioned by month; entry point to the lineage |
| step_executions | Each executed step: input/output snapshot, error, verdict, cost | The fine grained audit; partitioned like traces |
| claims | Typed facts (subject · predicate · object) + provenance + confidence + 3072d embedding | The team's memory; recallable by SQL and by similarity (DiskANN) |
| artifacts | What was produced (docs, videos, digests…), content addressable (sha256), with a status machine | pending_review → approved/rejected/expired; dedup by content |
| lineage_edges | Provenance edges between claims and artifacts | Recursive view (depth 12, cycle guard): "where did this come from?" |
| evaluators | Catalog of how to evaluate (deterministic / llm_judge / human) with a threshold | The per step verdicts come from here |
| ontology_overlays | Per vertical relabels over the same graph | Workspace A says "video reel," workspace B says "blueprint": same primitives |
| workspace_manifests | The navigable snapshot per workspace (Rule A) | Derived cache, rebuildable; manifest JSONB column as SSOT |
| chat_messages | The founder ↔ agent thread per conversation | The engine reads the last 20 to build the prompt; the app hydrates on reload |
| plan_drafts | Each request to Nova with its outcome (proposed/matched/rejected/launched) | Demand telemetry: the "couldn't do it" entries prioritize the roadmap |
| superskills | Promoted plans with a name, per workspace | Partial unique index against double promote; archive instead of delete |
+ in InsForge (identity, separate from the substrate): user_access (fail closed private beta gate) · magic_links (one time tokens with atomic consumption) · profile.app_state (the consumer's state) | ||
A detail a student should notice: workspace_id is on all of the workspace's data tables since migration 0001 (intents, traces, claims, artifacts, lineage_edges, manifests, chat, drafts, superskills); the definition tables (plans, steps) inherit it through their intent, and the catalog tables (operations, evaluators, overlays) are global by design. The data model has been multitenant from day one; the pending isolation is in the routing layer, not in the data.
workspace_id on every data table since 0001 (the catalog ones are global by design). When real multitenancy arrives, it is routing layer work, not a data migration with the product running.Measured against the repo on 2026-09-03, not estimated: the catalog is counted by importing OPERATION_CATALOG, and the tests by running the suites. Any of these numbers that goes stale is a bug in the document.
Multitenant routing (the data model is ready; the beta serves one workspace with a founder gate) · high availability and a formalized RTO (there is run durability and a nightly backup, but no replica and no documented restore drill) · proactive alerting with SLOs (there is observability, but nobody gets woken at 3am) · offline eval regression (the human gate already accumulates the labeled dataset; the harness in CI is missing). All four are acknowledged, classified, and committed to. The detail lives in the architecture red team.
To close the case study. The foundation of this system is brands any student recognizes: Postgres, nginx, Vercel, Cloudflare, Anthropic, Inngest. What is custom is a thin, typed, tested layer. The question that organized every decision was not "which technology is trendy?" but "what mature, cheap thing covers this without painting us into a corner?" And the only truly original bet, structured memory of human decisions, is exactly the piece no framework was selling. The full walkthrough of the product, with every step verified against production, lives in the operator's manual.
The Board responds
IRIS is scaffolding for a bottleneck this system does not have. Mirroring everything to RAM solves latency for high traffic synchronous workloads: hundreds of thousands of users, 10-15 tool calls per question. Agent Squad's load is the opposite: a few deep, asynchronous workflows, and a digest takes minutes by design. When you add a layer, ask yourself whether it survives the next model or whether you are adding it because today's model falls short.
The cost IRIS charges you is the one that hurts most in production: a second copy of your data via CDC is a second system to keep consistent, and consistency between two stores is exactly what fails on a Sunday at 3am, without you having anticipated it. Don't operate what you don't need to operate.
IRIS is a context solution for high traffic. Here the context is controlled by the catalog and the typed claims: you know exactly what goes into the LLM at each step, which is the truly hard problem. An honest nuance: if a high traffic synchronous agent ever shows up, the IRIS pattern fits inside an operation. Not today.
The intuition is correct and has already been acted on: there is an initial conversation where the system discovers what work you need, proposes the capabilities, and derives the team from there. The "3" was a presentation default, never a constraint. What comes next is your second half, the last mile: the agent producing with more autonomy, not the human steering step by step.
The squad was never the center of the design. The people are identity labels; the capability lives in the operations catalog. The planner composes from the catalog and never reads the squad, so the number of people emerges from the work and is not fixed in advance. Hardcoding the size would have been the mistake the student feared; the system avoids it by construction.
What you see is neither hardcode nor total autonomy: it is a router/state machine that generates the plan from natural language and validates it against a catalog, with guardrails. The team is a byproduct of which operations entered the plan. That is the difference between drawing a flow by hand and deriving it.
That he understand the model and how to give it tools and a goal, not that he marry a trendy framework the next model makes obsolete. The durable skill is knowing how much scaffolding to add and how much to delete.
That he learn to direct and delegate to agents. The bottleneck of the future is not the model's capability, it is the human's speed at assigning and reviewing. Whoever can orchestrate several agents at once performs like ten people.
It is still systems engineering. The hard work of agents is not the model: it is controlling what information enters each step. Systems fundamentals + "context engineering" don't expire with the next release.
That he learn to operate systems in production: observability, "how do I know this works and how do I fix it when it fails?" AI systems are nondeterministic; understanding how they behave when they break is a skill AI will not automate anytime soon.
On the verification ring: the robot that walks the entire product against production every night. To verify a single step you have to understand ALL the layers that cross it: screen, server, engine, database. It is the only seat from which you see the whole map, and with zero risk: it uses test users that self destruct. Test in prod, from day one.
The intern is the system's use case: a worker with energy and no formed judgment yet, just like the model. The same guardrails that contain the AI contain the intern, so they can do real work from week one, which is the only way to learn.
After the ring, give them a catalog Operation of their own: a deterministic piece from end to end. There they learn to control exactly what goes in and what comes out of a step, the core muscle.
An open source framework from Nous Research (Feb 2026): a personal, self hosted agent that lives in your messengers (CLI, Telegram, Discord, Slack), remembers across sessions, and self improves: after a complex task it writes a reusable "skill," without asking permission, and it connects to 200+ models. Its guardrails are infrastructure ones (container hardening, read only root), not approval of the actions. The question is not idle: it is among the fastest growing tools of 2026. hermes-agent.org
Hermes takes autonomy all the way: it self improves, runs long, takes the human out of the loop. For a personal assistant that is exactly what you want: the human is the bottleneck, remove them. Agent Squad puts a human gate in on purpose, because its effects land on someone else's business: a brief that gets sent to a client, not a script you run on your own machine. If your case is personal and tolerates autonomy, Hermes fits. If someone has to sign off on every deliverable, that gate is not bureaucracy.
The most honest difference: Hermes puts the guardrail in the container (so the agent doesn't break your machine); Agent Squad puts it on the action (so nothing with an effect happens without a signature) and adds provenance and SLOs. An agent that learns on its own and acts on its own in your messengers: how do you know what it learned, how do you audit it when it does something odd at 3am? For personal use, that opacity is your risk, and acceptable. For work that lands on clients, it is not.
Hermes auto creates skills (emergent orchestration): flexible for varied personal tasks. Agent Squad derives a typed graph validated against a catalog (explicit orchestration): controllable for repeatable, auditable business flows. And a point of humility: Hermes's "closed learning loop" and our promoted SuperSkills are the same idea, capturing a procedure to reuse it. The difference is who pulls the trigger: Hermes does it alone; we do it with a human who promotes.
Hermes is model agnostic (200+ models) and ships a harness: memory, a learning loop, 40+ tools. That is optionality and features, and also scaffolding the next model may make unnecessary. Agent Squad bets on a single frontier model and puts in minimal scaffolding + hard guardrails. Neither is "better": they are opposite bets. The question that matters is not which has more, but which of the two bets serves your problem.
From noise to audience
The substrate's market research was already deep. It measured the composite demand of each opportunity along three separate dimensions, never one disguised as another: the search for the angle, the market for the topic, and the social backing from Reddit and YouTube comments, attributed against a closed table so the model never writes a number it did not see. A dedicated pass tied down the evidence, and a ranking by Pareto fronts ordered the opportunities without adding apples to oranges.
And then a founder doubted a zero. The report said there were no valuable comments on YouTube, and he did not believe it: "I manually find cases that aren't praise". The doubt, pressed four times, uncovered real layers. First, four defects in how the engine discovered the videos: it searched for generic advice ("how to grow"), not for the problem videos ("why my channel isn't growing") where the audience asks questions. And a deeper finding: the lexical classifier could not tell a problem from a curiosity, because the difference was semantic; the clean signal was in another layer, in the provenance of the search, not in the title.
But the biggest discovery came with a video the founder brought in by hand: an instructor's masterclass, whose comments were personal stories. There the value was not demand. It was audience intelligence: the profile of who watches, their motivations, their barriers, and the gaps they reveal by telling their story. The engine was chasing it without seeing it. That is how a new pass was born, r14, which reads the comments the research has already sampled and extracts five honest things against a closed table: profile, motivations, barriers, gaps, and pain phrases as a literal verbatim quote, ready for the script.
In a real production run, in five languages, the report wrote down what used to be lost: a 70 year old creator who uses AI as an assistant, one who identifies as neurodivergent and loses the thread if they preproduce several videos, one who writes in French "j'espère que je vais y arriver avant que la maladie m'emporte". Real words from real people, written into the brief, ready to become content. The zero the founder did not believe was never the end: it was the tip of the thread.
The guardian between the engine and production
The substrate stopped depending on third parties for its own integration. The code and the CI became sovereign: they live on a box of its own, without asking anyone's permission. And with that move came the question that really matters: when the engine changes, what decides whether that change goes to production or not?
The end to end test suite had been marked "red" for days and left out of that decision. On closer inspection something different appeared: it wasn't broken, it had been green for five nights in a row. The traffic light was lying. What the gate's comment declared was corrected, and the real debt came into view, the only one left to pay: getting that suite to stop a bad deploy again, not just watch it go by.
The finding that organized everything was this: there is not one deployment, there are two. The one for the visible face and the one for the engine, each on its own path. And they are not validated the same way. The end to end tests run against a simulator of the substrate, so they tell the truth about the interface but not about the engine, which has its own suite. Treating them as one would have made a stumble in the face stop the engine for no reason. Each guardian watches its own door.
With the door made clear, the face's suite began blocking its deploy: the lightweight interface tests on every change, and the heavy tests of the live scene when the change touches that scene. The engine goes its own way, validated by its own suite. The substrate is no longer just built fast: it is published without a broken change slipping through, and that guarantee is proven by hand, one bite test at a time.
And the guardian was refined two more times. It stopped exposing its key: the token it publishes with no longer travels on the command line, where any process on the server could read it, but in the environment, out of sight. And it learned not to freeze: once the publishing tool hung on a network call and stayed stuck until the cap of the turn, forty five minutes, until it had to be killed by hand. Now each publishing step has a hard time limit and retries on its own, so a hang is cut off after a few minutes instead of jamming everything. A guardian that also doesn't leave its key in plain sight or fall asleep at the door.
From two engines
to one
The original engine (v1) lived in inngest/functions/execute-plan.ts. It received plan.compiled, walked the DAG from plans/steps, and kept a workflow suspended with step.waitForEvent when it hit a human gate. It worked, left useful traces, and proved the thesis. It also concentrated too much authority inside one function that is hard to reconstruct, inspect, and replace.
v2 changed the unit of truth. The compilador turns the template into a versioned graph of graph_nodes. A run lives in runs; what happened to it lives, in order, in run_events. A pure reducer decides transitions and effects; aplicar-evento persists them transactionally; the worker claims leases on work_items, executes operations, and returns new events. Derived state, a reproducible log, an explicit queue. This architecture won because it can be explained after a crash, not only while everything is healthy.
What it did. execute-plan walked the Plan, wrote steps and traces, retried operations, and suspended the workflow at the gates with step.waitForEvent.
Why it goes. Its effective state was spread between Inngest's history and the substrate's projections. Resuming, reconstructing, or changing the engine required understanding a live execution, not folding a stable log.
What it does. It compiles templates into immutable graphs, records every transition as an event, and moves work through a queue with leases. The reducer does no I/O: it declares. aplicar-evento persists. The worker executes.
What Inngest keeps. Durable event intake, crons, operational retries, and the tick that drives the worker. Inngest is still in the system; it stopped being the place where a run's state machine hides.
There is no longer a waitForEvent holding up the Plan. The ACTIVAR_GATE effect transactionally creates a record in artifact_approval_flows: the deliverable appears for review, the person approves and, when the protocol requires it, chooses an option; the executor acknowledges receipt and only then does the flow complete.
consumirHandoffV2 applies the decision to the run and chains the successors. The lock order is an invariant, not a recommendation: runs → artifact_approval_flows → work_items. It prevents approval, timeout, and worker from closing the same gate in incompatible orders.
Each flow freezes deadline_at, the fallback policy, and required_role. The barrido-gates-vencidos cron claims only the state it observed, via CAS: state = phase AND deadline_at <= now(). An approval that won the race cannot be overwritten by the timeout.
The gate is not the only barrier. An evaluator configured as blocking emits a hard stop verdict: if it fails, the downstream is not dispatched. The run does not turn a bad output into later work just because the worker is still alive.
“Going back to v1” sounded reversible, but it wasn't. v1 could not resume a v2 run, was not exercised with real traffic, and executed the template, not the compiled graph that v2 had authorized. Flipping the flag did not recover a run: it created another interpretation of the work.
With two engines, turning off the v2 worker could look like a precaution. With v2 as the only engine it is a footgun: it accepts runs, leaves work_items in ready, and nothing advances. That is why the default is now on; turning it off requires an explicit, observable decision.
The three operational resets follow the same conclusion: they normalize the routing to orchestration_state='active' and orchestration_engine='v2' without touching enabled. Resetting progress does not revoke permission or resurrect a retired engine.
Today · 2026-10-24
The system in one sentence: a 3D office on Vercel where every user has their own workspace and their own team of agents, on top of a substrate in a Hetzner box that turns each request into a typed graph, runs it on its own engine whose only authority is Postgres, reaches models only through the Vercel AI Gateway, records every step and every call and, in 25 of its 27 templates, waits for a person to approve before a deliverable counts as done.
What it is. At launch, the compiler turns the template into a graph (graphs and graph_nodes) and starts a run in runs. Everything that happens to it lands in run_events, in order and without gaps (29 event types, 9 run states), and every ready node enters the work_items queue with a lease. A worker takes up to 5 items per batch and runs them in parallel, authorizes each node again against the roster before running it, and retries a failed node up to 3 attempts in total.
What was left behind. Engine v2 has been the only one running since September 5, 2026, and seleccionarMotor only returns v2 or fails closed. The v1 executor is out of the Inngest registry; its file is still in the tree, marked @deprecated, and a test fails if anyone registers it again.
What it still does. 18 registered functions: 7 that respond to events (the declared intent, the asynchronous compose, FORGE and the canary, among others) and 11 crons. The key one is orquestacion-worker, which wakes the engine's worker every minute. None of them holds the state of an engine run: if Inngest stops, what was already decided stays in Postgres, but the work stops moving (see the lesson below).
What is not automated. Changing a function's id, triggers or concurrency requires a manual resync with Inngest: neither CI nor the automatic deploy does it.
What about a cron job in Postgres? The answer is in block I.
What it decides. One call to the model of the compose class resolves each request into one of seven typed outcomes: converse, gate, match, match_custom, plan, cannot and invalid. Compose never executes: it returns a proposal that the person launches. In production it is asynchronous (COMPOSE_ASYNC=on): it answers 202 with a task_id and the web polls the task until it finishes.
FORGE. It is no longer an outcome: it is a branch of cannot, switched on in production (FORGE_ENABLED=true), capped at 2 forges per workspace per hour, with human consent before spending and a second human gate before the capability is registered.
Operations. 75 id@version pairs over 71 ids (four live in two versions) and 71 registered handlers. The ones without a handler are deliberate: the engine resolves the two human operations (human_gate.approve and human_task), and slack.post and github.gist_create only run if FORGE registered that capability.
Templates and evaluators. 27 exported templates, plus two derived tails (recovery and repair); 25 end in a human gate, and the two without one are document-query-v1 and pricing-watch-v1. Nova has 10 curated workflows for match. The 12 evaluators run without a model: even the one declared llm_judge uses a deterministic implementation, and the judges that do use a model live outside evaluator.run.
What runs. Nova resolves a brief request with the content-brief-full workflow, and the graph version depends on research flags. In production the chain is switched on up to v13 and not v14, so the live graph is brief-synthesis-v13: 25 nodes, 43 edges, one evaluator and a gate with two phases. Every engine v2 run in the production database is on that graph.
What it does. It reads the audience; it researches YouTube and Reddit through ScrapeCreators, and search demand, the SERP and competitor metrics through DataForSEO; it synthesizes the research (r4), writes the brief (s4), evaluates it (s5), publishes it (s6) and leaves it at the gate (s7). The research budget in production is 1.50 USD per run.
How it gets approved. In phase 1 the person reviews the brief and can correct it inside the review; since migration 0126, the research synthesis too. Every version is kept in revisiones_de_brief and the original is never touched. In phase 2 they pick the angle. The deadline is 72 hours; if phase 2 expires, the system never picks for the person, and the only fallback that launch accepts is fail.
If something breaks. A run that dies before publishing has a recovery tail of 4 nodes (s4 to s7); a report that was already published has a repair tail of 10 nodes. Both require the owner's delegated principal. The report downloads as a PDF from the «Imprimir / guardar PDF» button (PR #167, October 10).
What changed. In June the beta served a single workspace; today every user has their own (user_home_workspaces, resolved with get_user_home()) and the API runs with MULTI_TENANT=on. On every request the web signs a delegation token (HMAC) with the user_id and the workspace, bound to the method, the route and the body, with a maximum life of 120 s. With DELEGATION_ENFORCE=on the API rejects anything without it and checks that the user is a member of the workspace.
What it is not. The 17 workspaces in production today belong, almost all of them, to test accounts that are reset often. This is isolation that is built and switched on, not traction.
What it does. Of the 85 tables in public in production, 70 have RLS enabled and 1 has it forced. The policy filters on app.workspace_id and fails loudly if it is missing; withWorkspace sets it on every transaction. The runtime uses the substrate_app role, without BYPASSRLS; lease reclaiming, maintenance, resets and the admin console have separate roles, each with its own URL.
Before running anything. The worker authorizes each node again against the current roster (autorizarEjecucion), and Presidio anonymizes documents before they reach the vectors; if Presidio is down, the original stays in quarantine.
Production reaches a model only through the Vercel AI Gateway, and not by configuration but by construction: outside development mode, the CLI, direct Anthropic API and Bedrock strategies throw an error; the service PATH has no Claude CLI; CI checks with static analysis that no production module invokes it, and LLM_API_FALLBACK=true prevents startup. The model is chosen by class:
| Class | Model | Used for |
|---|---|---|
| compose | spacexai/grok-4.6 | Nova's compose, FORGE, single task analysis and some research operations |
| discovery | anthropic/claude-opus-5 | Nova Discovery: proposes 2 or 3 agents during onboarding |
| cortesia | anthropic/claude-haiku-4.5 | Kickoff, the question after a rejection and the Nova panel in the admin console |
| standard | anthropic/claude-sonnet-4.6 | Chat, charter and most text operations; the lite and bulk classes use the same model |
| synthesis | anthropic/claude-sonnet-4.6 | Research synthesis (r4) |
| keywords | anthropic/claude-opus-4.8 | Keyword expansion (r0) |
| judge | anthropic/claude-opus-4.8 | Judges: document.query faithfulness and the board panel |
| content_brief | anthropic/claude-opus-5.5 | Pinned in production. Writing the brief (s4), with the v15_dual contract; the code's base profile is Sonnet 4.6 |
| voice | anthropic/claude-opus-4.8-fast | Pinned in production. The think step of the voice call; it is not a class, the VOICE_THINK_MODEL variable sets it |
Except for the two marked rows, these are the code defaults. The model_routing_policy table can change any class except content_brief, and its production rows were not surveyed. Jev, the router for node s4, runs in shadow: it decides and gets logged, but it does not change the model. Outside the LLMs, document.query reranks with Cohere rerank-v4.0-pro (active in production) and embeddings are local: multilingual-e5-small, 384 dimensions, inside the process.
What they see. OpenTelemetry exports to Honeycomb (service agentsquadtraces, environment prod), with one span per LLM attempt and per paid effect. Phoenix runs on the same server, in three containers, and receives metadata only, never prompts.
What went away. Langfuse was retired on August 22, 2026: its containers (web, worker, ClickHouse and MinIO) were removed. The code keeps a client that does nothing without a server.
What it stores. One row per model call: model, tokens, cost and outcome, never the prompt. Charges from research providers go to research_provider_attempts, and the timeout shadow (TIMEOUT_SHADOW=on) measures each attempt against the gateway in llm_attempt_shadow without cutting anything.
What it does not say yet. The code has estimates per call and caps, but no typical cost per run has been measured: it has to come from these tables, not from a number someone wrote down.
What it watches. The slo-alert.ts cron compares the SLO snapshot with its targets, warns if a metrics source fails, checks whether FORGE is stuck and runs the canary of the asynchronous engine; a dead canary and a v1 executor that wakes up again are the two critical alerts. It notifies through Telegram, with a cooldown of 6 hours for warnings and 15 minutes for critical alerts.
Known blind spots. learn-failures.ts (08:00) groups failures from step_executions, a table that engine v2 does not write, and two checks in slo-alert.ts read journald, where the API does not write. They are documented as limitations, not as coverage.
What runs. A daily pg_dump at 03:00 with a copy to Cloudflare R2; an incremental pgBackRest at 04:00 with continuous WAL to R2, which allows recovery to a point in time; and a restore drill every Sunday at 06:00. On October 24 both backups of the day were OK, and the last successful drill was on October 18 (99 tables).
What is missing. There is no hot replica: the RPO depends on the WAL (minutes) and on the daily dump.
What happened. From October 8 to October 24, 2026 the Inngest queue stopped moving in stages: first the orchestration worker, then the canary and finally the rest of the crons. The container showed as healthy, the app showed as connected, and the Inngest logs only showed publishing event. The canary's critical alerts reached Telegram for days, but nobody acted on them until someone checked the canary by hand.
How it is detected now. The signal was there: the canary of the asynchronous engine, one of the two critical alerts of the SLO cron. What is new is the procedure to confirm it without relying on the Inngest logs, which do not record executions: count the POST /api/inngest calls in the API log at two moments, and look at each function's queue in Redis (ZCARD, and the score of the first item, which is its scheduled time: if it is far behind and does not change between two reads, the queue is not moving).
How it is fixed. Before restarting, check the cron schedules: after a stall, overdue entries remain, and on restart each one catches up minute by minute. On October 24 that created 800 worker runs in 11 minutes, and the history grew 17 MB per minute with the disk at 99%. The procedure now is to stop Inngest, back up Redis, remove those overdue schedules and the pending runs that have no work, and only then start it. Redis is never wiped entirely, because the queue lives there.
What we did on October 25. The old history, 19.7 GB, was archived to Cloudflare R2, with its sha256 checked after downloading and decompressing it, and Inngest started with a new history that keeps only the function configuration. Rotation is now a command that is run by hand, with a short stop: it builds the new database and checks that the crons are still alive; if that check fails, it tries to go back to the original, and an interruption halfway through is repaired by hand with the same command. An hourly alert sends a Telegram message when the history passes 2 GB, about 6 days at the current rate.
What is still open. The likely cause of the stall, a SQLite history with no limit, is contained but not solved: it grows about 330 MB per day, and 85% of it comes from four crons that run every minute and almost never have work. The plan to harden Inngest is in block I.
Three guards were still looking at the v1 world: learn-failures.ts reads step_executions and also fails on its query; the live run guard of the automatic deploy counts running traces, which engine v2 does not write; and two checks in slo-alert.ts search journald. None of them fails loudly: they simply see nothing. They are now listed as known limitations in the TRD.
With the flags off, the code builds the brief with v3 and leaves FORGE off; production runs v13 with FORGE on. That is why the October documentation labels every statement as active in production, code default or unused, and this chapter spells it out wherever the code and production differ.
Migrations are applied by hand in production before merging the PR that uses them, and there is no version table; CI applies all of them, in order, on a test database. The Inngest resync is not automated either. The API deploy is: a cron every 3 minutes that only deploys when CI is green.
It is the question we get most often from people who read this document: "Inngest doesn't work well, you could replace it with a cron job in Postgres." We answered it with the code on main and with production reads from October 26, 2026, and put the answer through an adversarial review by two other models (Grok 4.6 and GPT-6 Astra).
Mostly, it is a clock. 11 functions have a cron (9 of them also accept an event) and 7 respond only to events; 12 of the 18 have a single step. The v2 engine worker runs its whole batch inside one step and with no retries: the durability of a run already lives in Postgres, with a lease, FOR UPDATE SKIP LOCKED, a heartbeat and a watchdog.
Where it really is a durable engine. handle-intent-declared memoizes its steps, so a retry does not repeat the ones that already finished (starting the run is also idempotent in Postgres); the shadow board reuses the model stages that Inngest already confirmed, although if the response is lost after the call, the stage can run again; and FORGE has two waits for human approval of up to 72 hours each, with step.waitForEvent.
It schedules SQL, not the work. pg_cron runs SQL inside the database and records those runs. The real work is TypeScript: model calls through the gateway, tools and video. Running it would need a TypeScript consumer that reads a queue, or calling the API over HTTP with pg_net, which this image does not ship. Either way, tracking, retries and concurrency for the work would still have to be built.
And it costs more than it looks. pg_cron 1.6 ships with the image but is not installed: installing it means changing shared_preload_libraries and restarting the production Postgres. Even then it would not cover the 7 event functions, the concurrency limits or the FORGE waits.
A job runner on Postgres. Really replacing Inngest would mean a TypeScript job runner on Postgres, such as graphile-worker or pg-boss: a queue with retries, LISTEN/NOTIFY to wake up instantly, and enqueuing in the same transaction as the state change. It fits the v2 engine.
What would have to be ported. Step memoization, the two FORGE waits of 72 hours, event idempotency and concurrency per function. It is a medium or large project, and in the last 30 days there was a single v2 run.
It caused October's worst incident. Inngest was stuck for 16 days with no errors in its logs (block H). Part of the pain comes from how we host it (a SQLite history with no retention, an unpinned image, crons that run every minute and are almost always empty, and a race with Redis when Docker restarts) and part from the integration (the HTTP connection with the API, a worker that nobody wakes up, and the manual resync). The SQLite history is the likely cause of the stall, still unconfirmed.
And a risk still to check. The API server closes idle connections after 255 seconds, and the log shows worker batches of up to 898 seconds inside a single Inngest call. We still have to check whether Inngest received those responses.
We keep Inngest and harden it. Pin the image version, start Inngest only after Redis finishes loading, wake the worker with an event when work is enqueued, move the four crons that almost never have work from every minute to every 5 minutes, and check the risk of long batches. Moving the history out of SQLite comes later and is reassessed with data: how fast it grows, whether write errors appear and whether rotation is still viable; sooner if the stall happens again. Every change is tested first on a rehearsal Inngest, and nothing has changed in production yet.
When the migration question reopens. If the stall comes back after the hardening, if a batch is cut off halfway with work already paid for, or when FORGE is rebuilt on the v2 engine gates. A date is not a criterion.
| Topic | May (cover) | Today |
|---|---|---|
| Primitives | 10 | 11: audience was added |
| Operations | 16 | 75 id@version pairs over 71 ids |
| Templates | 4 | 27, plus two derived tails for recovery and repair |
| Workflows | 4 in production | 10 curated workflows for Nova's match |
| Engine | Inngest walks the DAG | Engine v2 on Postgres; Inngest remains as durable intake, crons and the worker's heartbeat (18 functions) |
| Model | Sonnet 4.5 | Only through the Vercel AI Gateway, one model per class; Opus 5.5 writes the brief |
| Data | Postgres + pgvectorscale | Postgres 16 with pgvector 0.8.2 and vectorscale 0.9.0; 85 tables, 70 with RLS enabled; local embeddings with 384 dimensions |
| Cost | $0.06 for the smoke | No typical cost per run measured yet; a research cap of 1.50 USD per run, and every call is recorded in llm_calls |
| Tests | 28/28 passing | 880 test files in apps/api, 284 in the web app and 92 E2E specs; both deploys require green CI on Forgejo |
Sources: the production survey of October 24, 2026 (reads only) and, in the repository, docs/ARCHITECTURE.md, docs/CONCEPTS.md, docs/reverse-engineering/02-trd.md and docs/runbooks/alerting.md, updated to that date. Any figure in this chapter that goes stale is a bug in the document.
demo.digitalhubassist.ai, which does not resolve today: the buttons below will not respond until it is brought back up.