Persistence

An orchestrator managing autonomous coding agents will restart. Process crashes, OS updates, host reboots, deploys. The only question is how often. When it does, there are two versions of what happens next: one where the system picks up where it left off, and one where it doesn’t. The gap between those two versions is the gap between a production tool and a prototype.

The restart problem

Consider a Sortie instance managing ten concurrent agents across a project backlog. Three issues are in retry backoff: one at 40 seconds, one at 160 seconds, one at the five-minute cap. Two issues have already consumed four of their five allowed sessions. The orchestrator has accumulated eight hours of token usage data that feeds the Prometheus metrics powering your Grafana dashboards.

Now kill the process.

A stateless orchestrator loses all of this. Retry queues disappear. Backoff positions reset to zero, so the system immediately hammers the same failing issues at the base interval, creating a thundering herd on the tracker API right when you need stability. Session budget counters revert to zero, so a stuck issue that already burned four of its five max_sessions gets five more.

Token counters reset, and your Grafana dashboards show a cliff at 3 AM that has nothing to do with actual workload. Run history is gone: what the agent did, how many turns it took, how many tokens it burned. When a developer asks “what happened to PROJ-42 overnight?”, the answer is “we don’t know.”

None of this matters for a cron job that runs one agent against one issue. It matters enormously for a daemon that manages concurrent agents with retry logic, budget limits, and operational observability. For that use case, losing state on restart is a production incident, not an inconvenience.

Warm start vs cold start

The difference between stateful and stateless orchestration is clearest at startup time.

Cold start is what stateless orchestrators do. On restart, the system has zero memory of what happened. It polls the tracker, discovers issues, and starts dispatching from scratch. This works: the tracker is the durable record of what needs doing. But the tracker doesn’t store retry attempt counts, backoff timers, session budgets, or run history. Those lived in process memory and died with the process.

If the orchestrator was mid-retry on five issues with varying backoff delays, a cold start collapses all of them to immediate dispatch. Every pending retry fires at once. The tracker API gets hit with five simultaneous requests instead of five staggered ones. And for issues that were already approaching their max_sessions limit, the budget counter starts over.

Warm start is what Sortie does. On restart, Sortie opens its SQLite database and reconstructs state. Retry entries carry due_at timestamps: absolute points in time, not relative delays. Entries whose due_at has passed fire immediately (they were overdue anyway). Entries whose due_at is in the future get timers set for the correct remaining duration. The system recovers the exact backoff position for every pending retry, not an approximation.

Session budget checks query the run_history table. Completed sessions are rows in a database, so max_sessions enforcement works the same way before and after a restart. An issue that used four of its five sessions still has one left, not five.

Aggregate metrics (total tokens consumed, total dispatches, total worker exits by type) load from the database and resume accumulating. Your Prometheus counters don’t reset. Your dashboards don’t lie.

After loading persisted state, Sortie reconciles against the tracker. Are any persisted issues now in terminal states? A human might have closed a ticket while the orchestrator was down. Those get cleaned up. Then normal polling begins.

What is lost: running agent processes and their conversations. Agent subprocesses are OS processes. They die when the parent dies. Sortie rediscovers these issues through normal polling and re-dispatches them in a new agent session. The workspace directory is still on disk, so the new session starts from the files the previous one left. Prior commits, cached dependencies, and partial work are all intact.

What Sortie persists and why

The durable data falls into categories, each solving a specific problem that in-memory state cannot.

Retry entries. Each entry stores the issue ID, human-readable identifier, attempt number, a due_at timestamp in epoch milliseconds, the last error message, the agent session ID a continuation resumes, the agent kind, template, and rule name resolved at the issue’s first dispatch, and the issue’s place in its stage chain: the chain identifier, the previous stage, and that stage’s outcome.

Without the backoff timestamp, exponential backoff is a fiction: it only works within a single process lifetime. Kill the process, and every issue resets to attempt one. With persistence, a process restart at minute three of a five-minute backoff means the timer fires two minutes later.

Persisting the resolved agent, template, and stage data serves a different need: a retry or a reaction-driven continuation after a restart resumes with the routing the dispatch rules chose originally and renders the same stage data in its prompt, so the agent and prompt stay stable across the life of the claim.

Run history. One row per completed worker session: issue ID, identifier and display identifier, attempt number, exit type and error message, start and completion timestamps, workflow file, agent adapter used, the dispatch rule and prompt template that routed the run, the model and reasoning level the run was configured with beside the model the runtime reported, workspace path, turns completed, the self-review audit trail when self-review ran, the run’s place in its stage chain (the chain identifier, the previous stage and its outcome, and the stage the run’s exit targeted with the result of that hop), token counts (input, output, total, cache read, and cache write), a flag recording whether those counts are a measurement at all, the estimated cost of the run and how far it can be trusted, and a flag recording whether the agent ended the run by reporting itself blocked.

An agent that reports no usage produces an unmeasured run, which is a different thing from a run that spent nothing. A second count records the turns that reached the model without a figure covering them, because a run whose figures all arrived can still have spent more than they say. The two gaps are independent, and a total carrying either one is a floor rather than a figure.

This solves three problems. First, max_sessions budget enforcement: the count of rows for an issue is the count of sessions spent, durable across any number of restarts. Second, debugging: when an agent fails at 2 AM, the run history tells you what happened without digging through logs. Third, this is the raw material for cost attribution: every session records how many tokens it consumed.

Agent events. What the agent did during each run: the session it started, the tools it ran and how long they took, the turns it finished or failed with the token totals at each turn’s end, and the messages it sent. While a run is in flight its events live in memory. When it ends, Sortie writes them once, together with the run’s run history record, and the HTTP API serves them for at least 365 days after the run completed.

Run history says how a run ended; the events say how it got there. Without them, the answer to “why did the third attempt fail?” lives only in whatever logs were kept.

The cost is disk space. A run keeps at most 256 KiB of events, and all runs together at most 2 GiB, oldest runs pruned first. That cap counts the events themselves. Indexes and SQLite’s per-row overhead come on top, so a database at the cap measured 3.8 GB with events averaging 250 bytes. SQLite reuses the pages that pruning frees instead of returning them to the file system, so the file does not shrink on its own unless it is compacted with SQLite’s VACUUM. A long-lived instance should expect its database file to approach 4 GB.

Session metadata. The latest session details for each issue: session ID, token counts including cache writes, model, agent PID, timestamps, and exit type. This is the data the dashboard and API serve when you ask “what’s happening with PROJ-42 right now?” If the agent finished and the process restarted, the answer is still available because it’s in the database, not in a goroutine’s local variables.

Aggregate metrics. Cumulative counters (input, output, cache-read, and cache-write tokens, total dispatches, cumulative runtime, and a count of ended sessions whose usage was never recorded) stored under a single key and updated on every worker exit. When Sortie exposes these via Prometheus, a restart doesn’t create a false cliff in your time-series data. The counters pick up from their last persisted values.

Live stage passes. The stage passes that hold no retry entry: a running stage session and a budget hold, each with its chain, its rule, and the rule whose hop reached it. They exist for one reader. sortie stats runs as a separate process, and its stage funnels count work that has not finished, which only the database can show it. A queued retry needs no copy, because its retry entry already carries the chain and the stage.

These rows are the exception to this page’s rule that state exists to survive a restart. Sortie empties them at startup and restores nothing from them.

Reaction escalations. When a pull request reaction hands an issue to a person, Sortie stores the escalation: the issue and reaction kind, why it began, when, what it judged, how many tries it had used, and which pull request it watches. Ending an escalation clears the time and keeps the row, because what the escalation judged is what makes it apply its label and event once.

An escalation is a statement that Sortie has stopped and a person has to act. In memory alone, a restart would silently withdraw it: the reaction would rebuild with a full budget, dispatch another round of fixes against a pull request it had already given up on, and escalate again, posting the label and the comment a second time. Persisting it makes the restart invisible. The escalation stays until a fact Sortie observes ends it: the pull request resolves, the problem clears, or a person moves the issue out of the handoff state. The attempt counters of reactions that have not escalated stay in memory and start again after a restart.

SQLite as the right tool for this job

The choice of SQLite over alternatives like Postgres, Redis, or flat files is not incidental: it follows directly from the single-binary deployment model.

Zero ops. Sortie targets single-instance deployments on developer machines, CI servers, and small fleet nodes. Requiring a database server contradicts the philosophy that got you here: copy a file, run it, done. SQLite is an embedded library, not a server. The database is one file, stored in the same directory as your workflow file. There’s no connection string, no credentials, no network port.

Concurrent reads with single-writer semantics. SQLite’s Write-Ahead Logging (WAL) mode lets the dashboard query run history while the orchestrator writes a new retry entry. The orchestrator enforces single-writer access through one database connection: all writes serialize through that connection, matching the orchestrator’s own single-goroutine state mutation model. For write throughput measured in tens of transactions per minute, WAL mode is more than sufficient.

Forward-only migrations. Schema changes are numbered SQL files embedded in the binary. On startup, Sortie applies any unapplied migrations automatically, inside transactions. No migration tool, no manual steps. If the database has a schema version newer than the binary expects, startup fails. This prevents running an old binary against a database that a newer binary modified.

The migration sequence starts with the core tables: retry entries, run history, session metadata, aggregate metrics. It extends token tracking, then adds workflow file tracking to run history. Later migrations add run-history detail, self-review metadata, and a table that deduplicates reaction fingerprints across restarts.

Dispatch-rule routing columns record each claim’s frozen agent and template, so a retry or reaction-driven continuation keeps the same routing. Further migrations add the run-history token columns and the flag that separates a measured run from one whose agent reported no usage. Later still are the reset points for the consecutive handoff-absence sequence, the parked-issue table, and the table that dedupes the tracker comment announcing a per-issue budget hold across restarts.

A later migration adds a flag to session metadata recording whether that row’s API request count is a measurement at all, because the row’s existence says nothing about whether the counting path ever ran. That flag defaults the opposite way to its run-history sibling: session metadata holds one current-state row per issue that the issue’s next session overwrites, so an old row can afford to read unmeasured and heal itself on the next write, while an append-only run-history row that no later run will ever revisit cannot.

Another migration adds a dispatch identifier to session metadata, naming the dispatch whose session last wrote the row; a session-exit write clears it, so a row left over from a session that has already ended is never mistaken for one still running.

A later migration adds a cumulative count of ended sessions whose usage was never recorded to the aggregate metrics row, backfilled at upgrade time from existing run history and limited to runs recorded after the flag distinguishing a measured run from an unmeasured one existed; older rows read as measured and are not counted. From then on the count updates alongside the token counters on every worker exit.

A further migration gives run history a count of the run’s turns that reached the model without a figure covering them. It defaults to zero, so a row written before it reads as fully accounted even though nothing measured it: an append-only row that no later run revisits cannot now establish otherwise.

A later migration records what scheduled each queued retry and gives each parked issue its title and link, so a restart explains a retry the way it did before. Another keeps when Sortie first picked up the work, plus a queued retry’s model and its issue’s type, labels, and state. It fills the pick-up time from earlier agent runs on record and leaves it empty for work with none.

A later migration marks the first run dispatched after each stage hop. The run’s other columns cannot tell it apart: every later run in the stage the hop led to names the same previous stage, so only a mark set at dispatch separates the run a hop started from a later pick-up of the same stage. For history already on record, the migration marks the earliest run of each hop and leaves the later ones unmarked.

A later migration adds the table that holds reaction escalations, one row per issue and reaction kind. It starts empty, so an escalation that was standing when a database was upgraded is not restored, and the next escalation of its kind records one.

A later migration gives each run history record the dispatch ID of its run and adds the table that holds the runs’ agent events. It fills in nothing for runs already on record, so a run recorded before the upgrade has no dispatch ID and no events.

A later migration gives run history a flag for runs the agent ended by reporting itself blocked, and adds the table that holds live stage passes. A blocked run is recorded as succeeded with no error, so no other column tells it from a handoff. The flag defaults to not blocked, and a row written before it keeps that value, so a run in which the agent reported itself blocked, recorded before the upgrade as a plain success, is never read as a stop for a person. A run recorded with the status needs_person is not affected.

Operational simplicity. Backup the database by copying the file. Inspect it with sqlite3. Move to another host by copying the file. Stream continuous backups to S3 with Litestream. The operational model is as simple as the deployment model.

The trade-off is real. SQLite serializes all writes through a single connection. Two Sortie instances cannot safely share one database file. Multi-instance coordination is a non-goal, and this constraint is why. For the single-instance deployment target, write serialization is not a bottleneck. If multi-instance becomes necessary in an enterprise context, it would use a different persistence backend or a coordination layer above SQLite, not force SQLite into a role it wasn’t designed for.

The full decision rationale, including why Postgres, in-memory-only, and embedded key-value stores were rejected, is documented in the SQLite persistence decision record.

What durable state makes possible

Every surface that answers a question about past work reads these tables. None of them could exist over state that dies with the process.

Agents reading their own history. The workspace_history tool lets an agent query its previous run results on the issue it is working, so “what error did the last attempt hit here?” is a question the agent answers for itself rather than one that needs prompt engineering or manual context injection. Its sibling cost_budget reads the same rows to report how much of the token budget the issue has already spent. Both keep working after a restart because the rows do.

Cost attribution. Run history stores per-session token counts and whether the coding agent could measure them at all; session metadata stores the model. Each run’s cost is priced once, when its row is written, per model, under the rates in force then (your token_rates over the price table), and stored beside the tokens, with the source and version of the rates that priced it. A rate is a fact about the day the work ran, so a later edit of the rates prices the work that follows it and leaves the past as it was, and every surface that shows a finished run reads the same stored figure instead of each computing its own. sortie stats aggregates that history into spend by outcome, by coding agent, by dispatch rule, and by the other dimensions each run records, and the dashboard and Prometheus counters read the live side of the same figures. Per-issue chargeback and budget alerting are that data viewed differently, not new collection.

Audit trail. Every completed run records who (issue ID), what (exit type, turns, tokens), when (timestamps), and how (model, workflow file hash). This is the raw material for compliance evidence (SOC 2 audits, change logs, agent activity reports), even though Sortie itself does not ship an export or retention-policy surface for it today.

The design philosophy: collect the data once, at the source, so any governance surface built on top of it is a query away rather than a redesign. Persistence makes that possible.