Five studies that are changing how I think about AI in software engineering

SubscribeSign in

Five studies changing how I think about AI in software engineering<br>AI compressed the upstream work. What does that mean for everything downstream?

Brian Houck<br>Jul 10, 2026

21

Share

Welcome to the latest issue of Engineering Enablement, a weekly newsletter sharing research and perspectives on developer productivity.

Subscribe

🗓 Join me on July 23 for a readout of the upcoming State of AI Impact in Engineering: Q2 Report. We’ll discuss new findings from DX’s data on AI tool usage, spend, and impact across 500+ organizations. Register here.

Every once in a while, several independent papers arrive at roughly the same time and collectively tell a bigger story than any one of them does alone. This week, I’m sharing five recent papers that have significantly influenced how I’m thinking about AI and software engineering.<br>Each paper tackles a different question. Some measure the productivity impact of AI coding assistants. Others examine how those gains propagate through the software delivery process, explore what developers actually want from future AI systems, or reconsider the kinds of debt we should be paying attention to in an AI-assisted world.<br>Despite coming from different research groups and using very different methodologies, they all seem to be converging on the same underlying story.<br>AI is compressing the upstream work of software engineering. The more I sat with these papers, the less I found myself asking, “Is AI making developers faster?” and the more I found myself asking, “What happens after the code is written?” Are we actually shipping more value? Where do the new bottlenecks emerge? And what are the costs if understanding can’t keep pace with generation?<br>After reading these five papers, I came away with one overarching conclusion: we’re generating code faster than we’re generating the systems needed to safely understand, verify, and deliver it.<br>A quick note on disclosure: three of these papers come from people I know and work with extensively. None of the papers are mine.<br>Here they are, in the order I’d recommend reading them.<br>1. GitHub Copilot and Developer Productivity

Paper: Heilman, A., Kyllo, A., Murphy-Hill, E. GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis.<br>The first paper I want to highlight tackles the familiar question of whether GitHub Copilot makes developers more productive, but it does so with one of the more clever research designs I’ve seen.<br>Rather than simply comparing Copilot users to non-users (which are getting harder and harder to find), the authors control for Active Coding Time (i.e., how much time developers spend actively engaging with development tools) and examine how productivity changes within the same engineer over 43 weeks across a population of 16,223 developers.<br>The payoff of this design is that it compares engineers to themselves rather than to one another. Using that approach, the authors found that weeks with the highest Copilot usage were associated with ~40% more completed PRs per hour of coding time than weeks with no usage.

The relationship showed a clear dose-response pattern (a way to do a causal analysis, once everyone is already using the tools). More Copilot engagement was associated with more PR throughput, although the gains appeared to level off at very high usage.<br>The authors ran seven robustness and falsification tests to rule out alternative explanations (team-level effects, generic AI engagement, PR slicing, shifts toward easier work). The positive association remained remarkably consistent.<br>Interestingly, the gains were not concentrated in tiny PRs. The strongest effects were observed for larger PRs (7+ files), arguing against the idea that developers are simply breaking work into smaller units.<br>It’s a thoughtful analysis and shows that we’re not just coding more, we’re increasing coding efficiency as well. These findings anchor many of the studies that follow in this roundup.<br>2. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools

Paper: Demirer, M., Musolff, L., Yang, L. Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools.<br>The next paper I’m highlighting was published by the National Bureau of Economic Research. It analyzes AI adoption across 100,000+ GitHub developers and asks a more nuanced question than Heilman’s: when AI makes individual coding steps faster, how much of that gain actually survives all the way to shipped software?<br>The authors examine how AI productivity gains propagate through a hierarchy of software development: lines of code → files → commits → pull requests → projects/repos → releases.<br>They found that AI is clearly increasing coding activity, and the gains grow with each generation of tools. They estimate roughly +40% more commits from autocomplete, growing to +140% from interactive coding agents, and...