Make Every PR Prove Itself - by Michel Chatmajian

SubscribeSign in

Make Every PR Prove Itself<br>How we review 200+ agent-written pull requests a day with conclusive proof and a custom testing harness.

Michel Chatmajian<br>Jul 31, 2026

Share

Abnormal AI stops the cyberattacks that legacy tools miss. If your company still treats AI like a pilot program, you’re in the wrong place. AI-powered engineering is our default, and we’re hiring.

Agents at Abnormal generate and request review for over 200 pull requests a day, and the green CI checkmarks tell us little about whether they are actually safe to merge.<br>Agents write code faster than people can read or understand it, and the usual response we see is to make review faster: more AI reviewers, auto-triage the findings, auto-stamp the low-risk changes. We do some of that. It doesn’t touch the reason a diff is slow to review, which is that a diff shows you what the agent typed, not whether it works. The unit tests pass and still nothing in the PR shows the change running against a real service. So, often, the reviewer pulls the branch locally to run and inspect the remaining validation. This is slow, inefficient, and worst of all a huge context-switch for the engineer.<br>So we changed what shows up for review. Every PR Nora (our internal harness) opens has to prove itself: a screenshot from a running instance, the real API response with the expected fields, the log line showing the record processed, the new metric incrementing on a live request path. The reviewer checks evidence against the claim instead of simulating the program in their head.<br>That evidence isn’t easy to generate. Producing it means every coding agent gets its own real, near-production environment with real data, hundreds of times a day. In our stack, we call these testboxes .<br>Define the proof, then generate it

Whether a change is well-designed still matters to us, but the PR isn’t where we settle it. Design gets decided earlier, in the “plan” phase, before any code exists. By the time a PR opens, the open question is correctness, and correctness is what evidence can answer.<br>Proof takes two steps: define it, then produce it. Our company-shared plan skill writes a test plan into every spec and that plan lands on the PR before any code is written. The agent then configures testboxes to run pre-existing and just-in-time integration tests that produces exactly that evidence.

The testing plan is aligned with human reviewers before it’s sent off to a background agent to implement and test. This is a demo but real plans often contain dozens or more test scripts/narratives to run through.<br>We use the following template for every plan, separating verification into three phases:<br>## ... Phase N ...

... code ...

### Verification

#### Devbox Checks

[Devbox = local machine or Modal sandbox (interchangeable)]<br>...

#### Testbox Checks

[Testbox = real AWS credentials + live infrastructure (K8s pod)]<br>- [ ] [API/gRPC] New/changed endpoint or RPC returns expected status + body shape, plus the auth/negative case (real request against the live service)<br>- [ ] [UI] Before/after screenshot of the populated and empty states (needs a rendered frontend)<br>- [ ] [Kafka/consumer] Seed the input topic with representative messages on a declarative cluster, run the consumer, and assert what it drives<br>....

#### Manual Checks

[Manual = requires human judgment. If it involves running a command, it belongs above. May be empty. Each item should briefly note why devbox+testbox are insufficient.]

....

There is no single form of proof

The right evidence depends on what changed.

Take a UI change. An agent added a “per-cell version-drift indicator” to our deploy dashboard, a column flagging how far behind each cell is from the fleet’s target release. The browser agent brought the frontend up in a testbox against state in an isolated database and captured the before and after.

The “after” is a real running instance against distributed systems in our test environment — not mock data.<br>We’ve found interesting failures don’t look like binary exceptions. Early on, a UI proof came back as a screenshot of a page that returned HTTP 200 with the right title and then never mounted. Blank page, green comment, uploaded as evidence. With our latest testbox harness, the screenshot tool now reports what fraction of pixels differ from the most common color, plus a preview of the text that actually rendered.<br>Share one environment, override one service

The obvious version of this is a preview environment, and for a web app it mostly works. It stops working when the change is a Kafka consumer, or a metric, or a schema migration, and when you need a few hundred a day with isolated data rather than one per branch. We didn’t find something that would do that against our own service topology, so we built it.<br>A testbox is a real service orchestrator in near-production config, on a live endpoint, with its own private copy of every datastore it...