When agents make CI the bottleneck - by Frederik Dudzik
Human Systems
SubscribeSign in
Agentic AI overwhelmed CI, and test selection cut queueing from hours to minutes.<br>How coding agents pushed a one-person project into CI scaling problems
Frederik Dudzik<br>Aug 21, 2026
Share
The last time I had to think seriously about CI throughput, I was working on the CI/CD team at Shopify. I didn’t expect to run into the same kind of scaling problem on a hobby project with one engineer. But coding agents helped push one of my projects to roughly 500,000 lines of code, and I hit it much sooner than I expected.<br>Changes were arriving faster than CI could verify them. They also tended to be larger, and several could be ready during the roughly 20 minutes it took CI to verify one of them. Once one landed, the others often had to be rebased and run through CI again. Waiting created overlapping work, overlapping work created rebases, and those rebases created more CI work.<br>Thanks for reading Human Systems! Subscribe for free to receive new posts and support my work.
Subscribe
One engineer with coding agents was enough to make CI a bottleneck I had previously associated with much larger engineering organizations.<br>I addressed it by reducing how much CI each change triggered. By selecting only the tests a change could affect, many changes went from occupying the runner for around 20 minutes to taking a couple of minutes or skipping the suite entirely.<br>Why more CI capacity wasn’t the answer
At first, I paid for more CI capacity. I burned through GitHub Actions’ free minutes quickly, and the bill went from about $10 to $20 to $50 a month. It was still growing. I didn’t want CI costs to scale with the amount of work the agents produced.<br>I moved the workload to a dedicated runner on a generic VM provider and deliberately limited myself to one machine. One VM had enough total compute. A 20-minute suite was manageable overnight, when agents could take multiple turns without me waiting for each result. The problem was the 20-minute feedback loop during active development.<br>I could have added more runners, but I’d seen that approach at much greater scale at Shopify. More runners increase throughput, but every change still triggers the same amount of work. As the rate of changes grows, the required capacity grows with it.<br>I decided to reduce the work each change triggered instead.<br>Selecting only the tests a change can affect
The approach was one we had also used at Shopify: instead of running the entire test suite for every change, figure out which tests the change could affect and run only those.<br>I combine two pieces of information. Full test runs record which parts of the application each test uses. When a change enters CI, a TypeScript dependency graph shows which parts of the application the changed files can affect. CI combines those two sets of information to select the relevant tests.<br>Most of that work does not happen on every CI run. Scheduled full-suite runs collect the runtime information, while the TypeScript dependency graph takes only a few seconds to rebuild. A localized change may select only a handful of tests. A broadly shared change can still select most or all of the suite.<br>At Shopify, much of the codebase was Ruby, so we relied more heavily on runtime tracing to understand what each test touched. At that scale, collecting and processing those traces was expensive enough that we spent significant effort making it faster. TypeScript makes part of this much cheaper because the compiler can tell me how source files depend on one another without running the application.<br>The selection is deliberately conservative. If CI sees a changed source file that it cannot confidently connect to existing test coverage, it records that uncertainty instead of assuming the change is safe.<br>I still run the complete suite every night. Those runs also refresh the runtime information used for future test selections. This keeps the development loop fast while the nightly run provides broader coverage without blocking me.<br>Test selection does not guarantee that a skipped test could never fail. It avoids spending 20 minutes rerunning unrelated tests when CI has enough information to select a much smaller set.<br>Making test selection useful
Test selection only works well if the codebase has useful boundaries. If every part of the application depends on everything else, an accurate selector will still conclude that a small change can affect most of the test suite.<br>In this application, each page has a route file, and each group of backend operations has an API file. Larger parts of the application are also separated into packages. These boundaries give the selector useful connection points between a source change and the tests that cover it.<br>Some changes naturally cross those boundaries. Shared test setup, global styles, build configuration, and dependency changes can affect much of the application, so those changes still select most or all...