A Developer Survey Is Not a Measurement of Developer Productivity - Andrew WegnerSkip to content

Leadership Part 13 of 13 · Leadership Thoughts Mon 10 August 2026<br>A Developer Survey Is Not a Measurement of Developer Productivity<br>A year ago I cited METR's finding that AI made experienced developers 19% slower. They ran it again with an error bar so wide it no longer means anything. Here's what it means for the junior engineer problem I keep coming back to.

Article contents<br>Series Leadership Thoughts Part 13 of 13All posts by date →<br>Tags #job #leadership<br>Where the 19% came from¶<br>Almost exactly a year ago, I wrote about a randomized controlled trial by METR that measured open source developers 19% slower with AI than without. As part of my observation that AI assistants are decimating the "Junior Engineer" role at companies, this piece of information didn't help my view.<br>I've seen this in my own hiring, and touched on improvements my teams have made utilizing AI assistants in a post a few months ago. The story is the same though, while I've continued to hire for roles on my teams, none of those have been junior engineering roles.<br>I have a planned follow up to the Junior engineering crisis we are currently in, but let's get back to that METR study. METR started running it again in August 2025, and in February 2026 published what came back. Let's talk about this...<br>What the first study measured¶<br>Those 16 developers worked 246 tasks. The repositories averaged more than 22,000 stars and a million lines of code, and the developers had been contributing to them for years. Tooling was Cursor Pro with Claude 3.5 and 3.7 Sonnet.<br>Every one of those variables impacts the result. 16 developers is a small sample. A repository you've worked in for years is similar to a senior engineer at a company working on the same code base for years. They already hold the context an assistant would otherwise supply. Inline code suggestions in early 2025 are a different way of working than an agent running on its own for 20 minutes today.<br>What happened when they ran it again¶<br>The follow-up was a bigger study with 57 developers, 143 repositories, more than 800 tasks.<br>For the 10 developers who had also been in the original study, the estimated effect was an 18% slowdown, with a confidence interval running from 38% slower to 9% faster. For the 47 newly recruited developers, 4% slower, with an interval from 15% slower to 9% faster.<br>18% against the original 19%. The first result came back almost exactly the same. But the confidence interval around it changed. The original ran from 2% slower to 39% slower and excluded zero. Both of these new results cross it, which means the study can no longer separate the effect from nothing at all.<br>So this is not a retraction, and it isn't a failure to land on the same number. It's a measurement that came back the same size and stopped meaning anything. METR spent that post explaining why they no longer trust their own instrument.<br>Why METR stopped trusting the measurement¶<br>So what went wrong? METR lists six problems, names two of them as the important ones, and most of them follow from AI getting better over the past year to year and a half.<br>Three have the shape of a selection effect, and they all point the same direction. Developers started declining to participate once there was a chance they would be assigned to work without AI, so the study systematically missed the people most optimistic about it. Among those who did participate, 30% to 50% told METR they were holding back specific tasks because they didn't want to do those tasks without AI, so it systematically missed the work with the most to gain. And developers were less likely to finish a task at all once it was assigned to the no-AI condition. One of them completed none of theirs. Pay dropped from $150 an hour to $50 an hour over the same stretch, which METR believes contributed, though they put developers' expectations about AI ahead of money as the cause.<br>A fourth problem isn't selection at all, it's the clock. Developers told METR that timing a task got hard once agents were involved, because a developer waiting on an agent tends to go work on something unrelated while it runs.<br>Look at the direction of the first three. Refusing to work without AI, withholding the tasks where AI helps most, and abandoning the ones that landed in the wrong condition all push the measured result toward slowdown. The bias in the experiment becomes fairly obvious.<br>METR's own conclusion says:<br>Based on conversations with study participants, we believe it is likely that developers are more sped up from AI tools now - in early 2026 - compared to our estimates from early 2025. However, because of the selection effects in our experiment, our data is only very weak evidence for the size of this increase.

They are now redesigning around developer-level randomization, shorter experiments, and observational data.<br>What held up?¶<br>One result from the original study has held...