Reproducible, AI-driven large scale changes | Jean Barkhuysen
Large scale code changes (LSCs) are groups of code changes that all try to<br>accomplish the same goal, across a huge number of codebases.
For example, a common LSC is to migrate all codebases at a company from<br>libraryX@v1 to libraryX@v2, changing code along the way to account for<br>differences in API between v1 and v2. Due to their breadth, LSCs have high<br>cognition cost, owing to the high amount of contexts you have to reason about.<br>But also due to their breadth, they tend to have bounded problem spaces: the<br>wider a change has to be applied, the more generic the change must be. A change<br>that can’t be generically expressed is not an LSC, but instead a series of<br>individual changes.
AI agents are one way to attempt to tackle LSCs. Goals can be expressed<br>generically and handed to AIs as prompts to work on all target codebases. But,<br>when AI agents are set loose to perform LSCs, LSCs’ high cognition cost<br>compounds with AI non-determinism to result in low confidence in the changes<br>being applied, incur high costs, and produce results which are unauditable.
This article presents a different approach, based on a simple observation: most<br>changes that can be expressed generically can also be expressed mechanically. AI<br>agents explore and solve the LSC’s problem space in an evaluation flywheel, and<br>encode their results in a metaprogramming program which itself is used to<br>perform the LSC in a way that is deterministic, leads to higher confidence, is<br>cheap, and is as auditable as ordinary source code.
Context
At Google and now at Netflix, I find myself frequently having to perform large<br>scale changes to code: changes to hundreds or thousands of codebases. For<br>example, at Google, when I rewrote the internal Go status library (similar<br>external version<br>here) to support Go<br>1.13 error wrapping/unwrapping, Google’s<br>internal one-version-of-every-library + monorepo setup meant that I had to also<br>go make this work for ~56k Go programs. That involved making changes to hundreds<br>of programs with non-conforming / Hyrum’s-Law type oddities at the same time.
Most recently at Netflix, I’ve been working on problems requiring LSCs over our<br>thousands of Go projects. An example recent, long-running LSC is periodic<br>remediation of vulnerable dependencies.
Doing this kind of work in 2026 is dramatically faster with AI Agents, but<br>surprisingly there’s a good lesson to be learned from the old way of doing it<br>which can be paired with AI agents.
Before AI agents, we used to perform these changes by hand, and quickly learned<br>instead to build metaprogramming programs: programs which modified programs.
With AI agents, it’s natural to think to replace this with hordes of AI<br>sub-agents performing code change on each repository, with some shared prompt or<br>goal. However, it turns out to still be more advantageous to write a program for<br>the LSC, albeit now with an AI agent to explore the problem space and encode the<br>solution into that program.
The end result is a small program which, when run on all the target<br>repositories, solves the LSC goal in a deterministic, cheap, and auditable<br>manner.
Let’s take a look in closer detail.
Evaluation framework
This technique starts with an evaluation framework and a discovery / actuate /<br>evaluate loop. Using the vulnerability remediation project above, the evaluation<br>framework is largely self-evident. The goal is to remediate all Go repositories<br>of vulnerable dependencies. For each repository,
Go’s govulncheck<br>reports whether there are any vulnerabilities. (Goal: 0 fixable vulnerabilities)
The output and exit code of whatever program we write tells us whether we were<br>successful, and if not what failed. (Goal: no failures)
A PR’s CI/CD log tells us whether a PR to perform that remediation is valid,<br>and if not what failed. (Goal: valid PR)
This is all we need to begin our loop.
Discovery loop
With a sufficiently large enough set of repositories (which we have at Netflix,<br>or is generally available in GitHub), and enough compute, and enough tokens, and<br>a bounded problem space, we can solve the problem space and produce a single<br>solution.
We produce a solution to the problem space in the form of a program. That<br>program remediates vulnerabilities in whatever repository it is run in.
We encode the problem space, as we discover it, as a series of txtar<br>tests. That becomes the early<br>signal in our evaluation framework; it doubles up as a regression test<br>mechanism; and it triples up as audit documentation (This is how the program is<br>built to behave under scenario X, Y, Z).
As the agent explores the space, it encodes new situations it finds as txtar<br>tests, writes code to solve the expanded problem scope, verifies it first<br>against the txtar test and then by attempting to send PRs to its test repository<br>cohort, looking at failure and build logs, and re-assessing. When it reaches<br>quiescence, it expands its test cohort exponentially.
When we applied...