Make no assumptions. | Irrational Exuberance<br>I’ve recently been thinking a lot about the concept of “soil horizons”,<br>which is the idea that there are many distinct layers of soil, from topsoil all the way down<br>to bedrock, which all combine into a soil horizon.<br>Translating this idea into software, the ideal codebase would have a single uniform<br>“code layer”, but a surprisingly large percentage of production software has numerous, distinct<br>code layers as the leading architect shifted over time.<br>I’ve found this particularly true for software in problem-spaces with high essential complexity and low scale complexity,<br>where the purifying challenges of scaling never create enough pressure<br>to compact disjoint layers into a unified layer.<br>Codebases with the most code layers tend to be created by small teams working on complex domains<br>over a long period of time. In many companies this might be an identity, permissions or payments team:<br>stuff that’s permanently valuable, but usually not the central concern at any given time.<br>On such teams, there is often only one architect who understands the nuances of the domain well enough<br>to make tradeoffs. When that architect leaves, they are replaced by someone who aspires to operate<br>in the same code layer, but simply cannot because they lack enough context to do so.<br>As a result, that new replacement creates a new code layer, despite not intending to.<br>If the team runs through a handful of folks as the new team leads struggle, it’s easy to<br>end up with a complex code horizon very quickly.
The problem of messy code horizons is not a new one, and the general approach<br>to addressing them is the same one I wrote about seven years ago in<br>Reclaim unreasonable software,<br>but with the proliferation of coding and non-coding harnesses,<br>lately I’m running into the problem of messy code horizons more frequently.<br>Even more concerning, I’m seeing this problem expand from impacting code horizons into<br>impacting how organizations make decisions outside of software, e.g. the company’s general reasoning horizons.<br>When individuals or teams rely on LLMs to reason to conclusions, rather than using LLMs to explore or draft options,<br>it’s possible for even the most important decisions to be built on top of flawed reasoning layers underneath.<br>In the next section, I’ll develop the problem statement a bit about what I’m running into,<br>and then in the final section I’ll lay out the approaches that I am finding (moderately) effective to navigate that problem.<br>Messy reasoning horizons<br>If you give three enthusiastic engineers a problem, a new codebase, a coding harness, and self-approval rights,<br>it’s very easy to end up with three new soil horizons as their harnesses gleefully commit code.<br>However, in engineering we have a number of techniques to derisk this problem.<br>First, we have manual and automated code review, and second we increasingly have the<br>ability for the harnesses to operate off sufficiently clear instructions that they write new code<br>consistently with the existing code, even if the operator is unaware of what good looks like.<br>This is also true for code review, where coding harnesses<br>can drive consistency across pull requests even if the person (or harness) creating the pull requests<br>is not operating off the same shared context as the wider team.<br>Many codebases are not well-configured for this new reality, and those codebases are getting<br>worse at an accelerating rate as more harness and agent contributions get added.<br>Legacy codebases that reach a certain size before introducing these better practices are<br>easier to fix than before, but still require a lot<br>of work to fix.<br>That said, I’m confident that coding harnesses are going to substantially<br>improve the quality of code horizons over the next year or two as the way we configure harnesses improves.<br>That’s not the problem I’m worried about.<br>What I’m worried about is the application of harnesses to problems outside of writing software,<br>where there’s no static typing, linting, or unit tests to validate the output.<br>Let me provide a very recent example from my own work that highlights this problem:<br>I wanted to understand how our incidents were trending over time.<br>So I pulled data via an MCP, and the analysis was unintuitive to me, in particular<br>I thought we were having more Data related incidents than the results reflected.<br>I had to look at the incidents in Slack, then the results in our incident tool,<br>and understand why the two conflicted. After a bit, I recognized the results in our incident tool<br>were only showing incidents that properly tagged a team when the alert was triggered, so it was omitting about half the relevant incidents.<br>After having the agent manually tag the incidents without team assignments, the data<br>made a lot more sense. After recognizing the issue, it was trivial to fix.<br>However, if I had simply accepted the...