The Half We Don't Measure | Rishi Goomar

Skip to contentSearching...<br>No results found

Ask most companies how AI is going and you'll get a usage number back. Seats filled, tokens burned, percentage of engineers active in the tool, PRs opened. Those are all real numbers. What they don't tell you is whether any customer ended up better off.<br>That's the gap I want to talk about. If you're going to incentivize AI use, the thing worth incentivizing is a better outcome for the customer. Usually that shows up as speed. It can also show up as a lower defect rate, a system that holds up under load, or someone catching a problem in review that would otherwise have shipped. Everything else we're measuring is a proxy, and we mostly picked proxies that sit at the wrong end of the pipeline.<br>Throughput was a good signal once<br>Attach a number to someone's performance review and people start managing the number instead of the thing it was supposed to describe. Usage metrics were genuinely useful right up until they landed on a dashboard. Back when using AI was voluntary, someone with high usage was just someone excited about the tool. Once a manager could see the number, it started measuring something closer to willingness to look like you're on board.<br>PR throughput is the more interesting case, because there's real research behind it. It's been part of validated delivery work for over a decade and it does correlate with organizational outcomes. [1]<br>The catch is that research was done when writing code was the bottleneck, and the bottleneck has moved. Pair a throughput target with an AI usage push and you've told people to generate more code, which they'll happily do. Whether that gets a customer anything is a separate question, and sometimes the answer is that it costs you, because every line you merge is a line somebody now owns.<br>Where the incidents come from<br>This part tends to get left out of AI adoption conversations.<br>Two years of engineering telemetry covering 22,000 developers across more than 4,000 teams found real output gains under heavy AI adoption. Task completion up 34%, epics completed per developer up 66%. Those came out of telemetry rather than a survey, which makes them about the strongest evidence I've seen that this stuff works.<br>The same dataset has another side to it. Incidents per pull request rose 242.7%. Bugs per developer rose 54%. Median time in PR review rose 441%, and 31% more PRs merged with no review at all. [2]<br>That last number explains a lot of the others. Code generation got faster than review, testing, and deployment could absorb, so the reviewing stopped happening. AI-generated code is also unusually good at surviving the review that does happen. It reads cleanly, it's well formatted, and the comments are plausible. It passes the sniff test in a PR and then fails against the parts of your system the model never saw. That makes it easy to trust enough to ship without reading line by line, and you find out in production.<br>The cost of that lands on whoever's on call, who usually isn't the person who merged it. Which makes it an incentive problem. If the person generating the code gets credit for throughput while someone else absorbs the incidents, you've pulled the reward apart from the consequence, and you should expect exactly what you get.<br>Somebody has to maintain all of this<br>Incidents show up within days. The maintenance cost takes years to fully arrive, which is part of why it's so easy to ignore.<br>Research covering hundreds of millions of real code changes shows duplicated code blocks climbing every year since 2023, now at the highest level on record. Over the same period refactoring collapsed. The share of moved, or refactored, lines fell from 21% in 2022 to 3.8% so far in 2026, while copy and paste rose from 9.4% to 15.7%. Developers are now roughly five times more likely to duplicate than refactor, an inversion of the two-to-one preference for refactoring that held before these tools showed up. [3] AI adoption isn't the only thing that changed in that window, so I'd hold the causation loosely.<br>Every copy of a block is one more place a future change has to reach, and tracking them all down falls to whoever touches that area next, usually someone who wasn't there when it was written. Multiply that across a codebase that's growing faster than anyone can read it.<br>It shows up in capacity. An analysis of over a million PRs across thousands of engineering organizations put reactive work, meaning bugs plus maintenance, at 44% of capacity for the median org, with reverts growing faster than PR volume. [4]<br>You can argue with any specific number in here, and you should discount the ones coming from vendors who sell into this problem. The direction is harder to wave off, since it shows up across datasets that don't share a methodology. We built services faster than we built the capacity to run them.<br>So count the whole cost before you call something a win. A feature shipped in half the time that...