Why Software Requirements Get Easier in an AI Economy

Structure and Guarantees

SubscribeSign in

Why Software Requirements Get Easier in an AI Economy<br>Abstraction boundaries pay off

Adam Chlipala<br>Jun 23, 2026

Share

One of the best protections, in principle, against AI systems going off the rails during recursive self-improvement is formal verification, where we prove mathematically that systems meet formal requirements. A natural obstacle is being sure to write out in enough detail what rules we want to enforce, such descriptions being called specifications. An increasing role for AI in the economy can, on the one hand, increase the number of computer systems for which careful enforcement of rules is critical. However, there is a sense, perhaps ironic, in which the job of spelling out the right rules, of writing good formal specifications, becomes easier. I have made the case that, as AI proliferates, it makes sense to design the economy to include bubbles where AI interacts only with other AI, and I sketched how specifying user interfaces then becomes easier.<br>I will now make the case that the broader challenge of settling on system requirements also becomes fundamentally easier. The basic advantage comes from whose intentions need to be formalized in the requirements. Getting requirements out of squishy humans can be arbitrarily challenging, but what happens when the source of new requirements is overwhelmingly other programs already in production?<br>Thanks for reading Structure and Guarantees! Subscribe for free to receive new posts and support my work.

Subscribe

The Hard Part of Software Engineering

Even before AI coding assistants changed a software engineer’s mix of time spent on different activities, it was a truism in the field that the overwhelmingly time-consuming activity was understanding what users really want, or requirements-gathering. With recent automation of so much of routine coding, we might expect that the centrality of requirements-gathering will only increase. I want to suggest that, leaning even more into what will be possible with automation, we will ironically see requirements-gathering as we know it decrease in importance. To make that case, I should first review the conventional wisdom.<br>When programmable computers were first developed, the details of who did what to get them programmed were no doubt a little chaotic. Soon enough, though, around the 1960s, a split developed between analysts and programmers. The former had the job of understanding requirements and translating them into relatively unambiguous notation like flowcharts, which programmers could then translate into code. My own formal programming education involved just the slightest brush with flowcharting before that trend fully evaporated, but, in the course of a summer internship, I do remember being advised by a long-time technical employee of a large corporation that “programmer” was a poor choice of career path, representing a kind of cognitive underclass taking orders from analysts. (Perhaps developments in AI coding assistance make that advice more prescient than I realized at the time!)<br>The discipline of software engineering developed orthodoxies like the waterfall model, where teams are very careful to go through many iterations of written requirements before beginning the expensive process of programming. Writing the requirements is not just a process internal to a team of software specialists. Arguably most important is interfacing with the intended users of a program, to understand fully what functionality will satisfy them. One conversation is rarely enough. Instead, the intended users must be shown repeated revisions of a requirements document, to help them think more abstractly about the full range of relevant scenarios.<br>Eventually, waterfall went out of style in favor of agile methods, which focus on getting to working programs more quickly, so that users can give more-informed feedback. The problem is that even the software experts find it hard to enumerate all relevant scenarios for a complex program. Many scenarios only become clear as critical by relying on the expertise of the users, yet those users are not trained in the kind of abstract and systematic thinking needed to map out all relevant usage flows, with their steps and subtleties. Eliciting such information to guide a product roadmap remains so challenging today as to motivate the specialty of product management.<br>The challenge of learning what solution is really best for users is often explained using a quote attributed (perhaps apocryphally) to Henry Ford, about how if he had asked his customer base what they wanted in transportation, they would have asked for faster horses, not the cars that he eventually gave them. There are at least two separate problems. First, a user with a vague idea of the fundamental purpose of a piece of software may have trouble explaining that purpose in enough detail. Second, a user without an...