How we built the delivery evidence chain · AppaloftSkip to contentAI agents can write an app in ten minutes. Proving what actually shipped is a harder, older problem. This is how we built Appaloft’s evidence chain — and where we deliberately stopped.

Everyone building with coding agents hits the same wall. The agent produces something that works on your machine. Then someone — usually you — copies files to a server, restarts a process, eyeballs a log line, and declares victory. Two weeks later, when production is serving the wrong build at 2am, nobody can answer the simplest question in software delivery: what exactly is running, and who approved it?

This problem predates AI. But agents make it acute, for two reasons:

Throughput. A human writes one deployable change at a time and usually remembers what was in it. An agent can produce twenty candidates in an afternoon. Your deployment memory does not scale.

Trust. The moment you let an agent drive deployment directly — hand it SSH keys, a Docker socket, cloud credentials — you’ve given production access to a stochastic process with a confident tone.

Appaloft is an open-source (Apache-2.0) delivery platform built around one idea: promotion must be explicit, and delivery must leave evidence. Not “the agent says it deployed.” Verifiable observations, recorded at each step, that link the exact artifact you approved to the workload actually serving traffic.

We call this the delivery evidence chain. This post is about how it works, mechanically — and about the claims we deliberately do not make.

What an evidence chain is (and isn’t)

Let’s get the disclaimer out of the way first, because it’s a design principle, not legal cover: the evidence chain is not formal verification, not a correctness proof, not a security certification. It is a chain of custody. It answers:

Which exact bytes were frozen as a candidate?

Who — what kind of principal — approved promoting them, and when?

After deployment, does observed reality (the running container, its configuration, the route serving traffic) match what was approved?

If any link in that chain breaks, the system says so, explicitly, with a reason code. Absent or unavailable evidence can never silently pass a check. That fail-closed property is the whole point.

The chain has four links: freeze → preview → promote → verify .

Link 1: Freezing the workspace into a content-addressed artifact

A coding agent works in a sandbox — a gVisor-isolated container with a deny-by-default network policy. When its work looks good, you freeze the workspace into a source artifact.

The obvious implementation is “tar the directory, hash the tarball.” We didn’t do that. The artifact digest is a SHA-256 over a canonical JSON manifest — a sorted, deduplicated list of {path, sha256-of-file-bytes, sizeBytes} entries. The archive itself (a plain uncompressed ZIP) is just a transport. Identity lives in the manifest.

Why? Because manifest-of-hashes gives us properties a tarball hash can’t:

Dedupe that’s meaningful. We can look up findArtifactByDigest and return the existing artifact when the agent’s “new” output is byte-identical to a previous one — including when two freezes race each other (the loser of the race deletes its copy; the caller gets the winner’s descriptor either way).

Mutation detection. After reading each file during capture, we re-check its size. If the workspace changed mid-freeze, capture fails: “Source Artifact changed during capture.”

Path safety. Every entry is validated against the source root; symlinks and .. escapes are rejected outright.

Two more design decisions worth mentioning:

Freezing requires quiescence. Capture is rejected if any agent run is active in that sandbox. You cannot freeze a workspace mid-edit. This sounds obvious; enforcing it in the state machine (rather than documenting it) is what makes it true.

Secret scanning happens at freeze time, fail-closed. Before anything becomes immutable, we scan for a filename denylist (.env* except documented templates, id_rsa*, credentials.json) and content patterns (PEM private keys, AWS/GitHub/OpenAI token shapes, password = …-style assignments). If it looks like a secret, the freeze fails. Secrets cannot enter an immutable candidate — because everything downstream (preview, promotion, deployment) will treat that artifact as trustworthy-by-identity.

Link 2: Preview what you froze, exactly

A frozen artifact feeds a candidate preview: a temporary, expiring URL serving that exact build. Every preview response carries an x-appaloft-artifact-digest header, and the control plane verifies the gateway echoes the same previewId and artifactDigest it expects — a preview that can’t prove which artifact it’s serving is treated as unverified evidence and blocks promotion.

Preview tokens are HMAC-hashed, compared with timingSafeEqual, revocable, and GET/HEAD only. Small TCB, deliberately: the preview gateway is a standalone service with a few hundred lines of...