Software factories are distributed systems — Stephanie Jarmak<br>Skip to content

Every abstraction here is taught through Gas City, a software factory SDK I run and help maintain.

An agent in my fleet fixed a scoring-integrity bug (yay!). It made the change in its own branch, added a regression test, ran the suite, passed automated review, recorded a verdict of pass, and closed the work item. Every step reported success, and every step really did succeed.

Nothing merged the branch (not yay).

The commits sat there, three ahead of main, with the new test present on the branch and absent from the codebase everything else was reading. This was unfortunately an embarrassingly common occurrence for me. Elsewhere in the same rig (basically a repo plus a work-items database in Beads), audits kept reading main, finding bugs whose fixes had already been written, tested, reviewed, and closed on branches nobody merged, and filing them again.

The working code existed, but the system never folded it in.

Some of this amounts to a game of Agent Telephone. An agent will do what you tell it to, sometimes, except when it doesn’t, or when it helpfully decides you meant something adjacent. But that wasn’t the problem here. The agent did the work it was asked to do and reported accurately on what it had done, and the missing step was outside the agent entirely. Nothing owned the transition between producing a valid change and making that change part of the codebase.

That meant a closed work item and merged code were allowed to stand in for the same fact when they are very much not the same fact.

One layer observed something, another layer treated it as authority, and nearly every failure below is a version of that. Agents fail on their own too, and later sections have plenty of them: workers skipping a check the protocol told them to run, closing work the instructions said to leave open. But those aren’t failures you fix by writing a better instruction, which is the actual problem. A prompt can’t hold an invariant. So the factory around the agent has to keep track of who is allowed to act, whether an effect has already happened, what artifact was actually verified, when work is really complete, and what to do when part of that sequence dies halfway through. None of those guarantees come from the model or its toolkits. They also do not belong uniquely to a workflow engine, queue, or agent runtime. They emerge from how all of those pieces interact.

An old problem with a new worker

Not every agent deployment needs a factory around it. A local assistant that reads a repository, proposes a patch in an interactive session, and exits has one process, one human, and very little durable coordination state. If it dies, the human restarts it and mostly loses some minor convenience.

The failure model changes once 1) the work has to outlive the process doing it, 2) multiple workers can act concurrently on shared or versioned state, 3) components can fail independently, 4) external systems commit effects asynchronously, or 5) verification and publication happen in separate places. At that point you have acquired the usual distributed-systems problems whether or not you’ve thought to wrap your head around it that way: stale authority, duplicate effects, lost updates, split-brain records, and partial failure, to name a few.

Thinking about software factories this way isn’t anything new. Osterweil argued in 1987 that software processes are software too. Choi and Scacchi described “the software infrastructure for a distributed system factory” in 1991, treating the coordination plane as something that had to be engineered in its own right. The CNCF’s Secure Software Factory reference architecture supplies much of the contemporary vocabulary.

What autonomous agents change are the characteristics of the worker. The mechanisms in this essay are not new. Fencing, leases, idempotency, reconciliation, and conditional writes have decades of distributed-systems history behind them. What surprised me was how little scale it took before I needed them. You can get stale authority, duplicate executors, conflicting effects, and split-brain records with three coding agents sharing one repository. “Distributed system” starts sounding grandiose right up until one worker dies, its child keeps editing, and the retry starts another one.

The interesting boundary is not fleet size. It is the moment workers can act independently on durable or shared state while their supervisors, observations, and external effects can fail separately.

Those older systems mostly coordinated deterministic tools and human developers who could, at least in principle, be asked what they had done and why (instead of “idk Claude said to do this,” unless Claude happened to be that one guy hoarding the company COBOL knowledge). A compiler does not confidently explain that it compiled the program when no binary exists. An agent can absolutely tell you it completed a task whose...