Turn the Lights On Before the Factory Starts — Next Wave of Tech

▶ NOW PLAYING · CH 03<br>Lights on<br>Demo Reels · JUL 22, 2026

▶ CH 03

TAPE №010<br>Demo Reels·JUL 22, 2026·11 MIN<br>Turn the Lights On Before the Factory Starts<br>What happened when a team of eight moved peer review from generated code to the plans humans wrote before it.

We’re a team of eight, and over the past couple of months we’ve reached plan 660 in our own software factory.

Every change starts with a plan reviewed by another person. An agent then builds it, while automated reviewers check the code for bugs and compare it with what the team approved. Some changes merge without an engineer reading the full diff.

In Software Factories, Light and Dark, Addy Osmani calls those implementation loops dark: machines build the code, other machines verify it, and no person reads the output before it ships.

Across a sample of 234 reviewed plans, we sent 17% back for changes. That is roughly 2.8 times the rate at which we previously requested changes on pull requests. Yet we’ve seen 18–26% of the resulting pull requests receive no recorded human review on GitHub.

Human judgment still sits at the entrance to the factory. We keep the lights on while people decide what should be built, then let individual implementation loops operate with different levels of autonomy.

The question is what a loop must prove before we let it run with the lights off.

Our review gate was in the wrong place

We published AI Is Not Your Peer because code review had started to feel like a different job.

When a person writes code, the pull request contains choices they made. You can ask why they chose a particular boundary or what they expect to happen on failure. The author can explain themselves, disagree with you and occasionally change your mind.

An agent-generated diff is a messier artifact. It contains the decisions a person made, mixed with hundreds of mechanical choices the agent made on their behalf.

I experienced this from the slightly unusual position of a product manager opening code changes.

The comments engineers left on my pull requests were generally reasonable. They just concerned decisions I had never made and often did not care about: which internal pattern the agent chose, how it organized a package, or whether another library would have been more idiomatic.

The engineers were doing what code review asked of them. They read the diff and formed an opinion about the implementation. Meanwhile, the product behavior I cared about was buried somewhere inside it.

The timing made those conversations worse. A pull request opened in the morning might receive feedback the following day, when I was already deep in customer calls or another piece of work. A question that took five minutes to answer while the problem was fresh could cost half an hour once I had to reconstruct the whole thing.

We were spending our most expensive conversations on an artifact that fused human intent with machine implementation.

So we moved the conversation.

We moved peer review before the code

We started writing plans that contained the business context, the proposed behavior, the systems affected and the evidence that would show the work was complete.

The important implementation choices went in there too. If the team cared which library the agent used, how a migration worked or what happened during failure, that decision had to be visible in the plan. Anything left unspecified was something we were explicitly allowing the implementing agent to decide.

Another person then reviewed that plan.

This is where peer review now happens for us. A human who knows the system pushes on the approach and its consequences. They catch the cases the author missed. The author can defend the decision because it is still their decision.

Only after that review does an agent start implementing.

The change sounds almost administrative. We swapped one review artifact for another. What surprised us was how much the timing altered the cost of disagreement.

Feedback became roughly 22 times cheaper

We measured how much a round of feedback added to delivery.

During our pull-request-review era, work that received a change request took a median 15.9 hours longer to merge than work that did not. That comparison covered 13 pull requests with feedback and 194 without it.

During the plan-review era, a round of feedback added a median 0.7 hours . That sample contained 11 plans sent back and nine accepted without a feedback round, each linked to its resulting pull request.

This does not mean plans are 22 times faster to read. It means changing your mind is much cheaper before an agent has built the decision.

The samples are small, especially on the plan side. This is one team working in one repository. It is not a controlled experiment or a promise that another team will see the same result. The size of the difference still changed how I think about review.

When feedback was expensive, reviewers used it...