Do We No Longer Care About the Code? - Schalk Neethling - Open Web EngineerSkip to navigation<br>Skip to main content<br>Colour themeSystemLightDark

I read The Machine Kept the Code by Stephan Huber, and watched Kent C. Dodds on why we should stop reviewing diffs and start reviewing systems, and between them they finally convinced me of what I have been toying with in my head. It is not that we do not care about the code anymore; we care about it differently. The system, the end-user experience, and the product are now taking center stage.

As engineers, we are moving up one level, or zooming out, depending on which you prefer, to consider the system that delivers the product that solves a real user problem. We also get to collaborate more closely than ever with the rest of the team on that experience, from product owners and designers to UX practitioners and stakeholders. Many teams have worked this way for years. What changes is the depth, because the specification is now the thing being handed over, and it is a document everyone can read and correct.

We spend our time on the specification, we sweat the details, we ensure we clearly define both success and failure, and then we hand it over to the agent to implement. Once shipped to a testing environment, we test to ensure our expectations are met on every front: performance, accessibility, user experience, the features we introduced or improved, the business goals, and our definitions of success and failure.

The standard we hold ourselves to should rise accordingly, and it should rise across the whole of what we make. A great deal of what we used to accept was accepted because the effort was real and the time was short. When implementation is no longer the expensive part, that defense weakens everywhere at once.

Accessibility and performance are two of the pillars, and they are necessary, but they are not sufficient. Each piece can be accessible and fast and the end result still useless to the person trying to get something done. A product can meet every success criterion and still offer an experience nobody wants to return to. The higher standard has to cover whether the thing is worth using at all, which means it covers the research, the design, the copy, the error states, and the decision about what to build in the first place.

The Two Questions

When we encounter problems at any point during this new software development lifecycle, we start by asking two familiar questions:

How did this happen?

How can we prevent this from happening again?

We have had to ask and answer them many times during postmortems in our careers. What has changed is where these problems originate and when we catch them. Problems will still reach production. The goal is to catch more of them earlier each time, and to let each one strengthen the checks that come before it.

When the Answer Is the Specification

If the answer points at the specification, we need to understand where it came up short or where we were too ambiguous. Ambiguity is what really bites when working with AI agents. We must be clear in our language, ensure there is a common understanding, and provide a clear statement of what success and failure mean.

Defining failure deserves as much attention as defining success, and it covers two different things. There are the behaviors that would be wrong: the states the system must not enter, the inputs it must reject, what is explicitly out of scope. Then there are the failure modes: what happens when a request times out, when the network is unavailable, when the data is malformed, and what the user sees in each case. A specification that describes only the desired outcome leaves the agent to invent both, and it will invent them plausibly, which is more difficult to catch than inventing them badly.

State diagrams are one of the most direct ways to remove this kind of ambiguity. Enumerating the states the system can occupy, and the transitions that are legal between them, turns a set of assumptions into a document. Every transition that is not drawn is a transition that should not happen, which converts an unstated boundary into a stated one. Written in a text format such as Mermaid, the diagram lives in the repository next to the specification, which means the same artifact is readable by the team, by the client, and by the agent implementing against it. It also gives the tests something concrete to assert against, because an illegal transition is a testable condition rather than a matter of judgment. A test that asserts an illegal transition cannot occur is a clear example of the deterministic checks we must build into our systems.

When the Answer Is the Implementation

The state diagram is a small example of a larger pattern. If the problem is the implementation rather than the specification, we must clearly document the defects, and from there we define the deterministic guardrails and tooling that catch the same defect the next time it appears and report it to...