There Is No Reward Function for Meaning | Aaron AngThere Is No Reward Function for Meaning<br>27 Jun, 2026<br>We Only Celebrate Progress Once<br>A century ago, sending word to another continent meant waiting weeks for a ship to cross the ocean. Today, we complain when a video call lags for half a second. For some reason, we celebrate a breakthrough once, then it quietly becomes the baseline.<br>Smartphones condensed cameras, maps, encyclopedias, music players, and libraries into devices that fit in our pockets, yet we&rsquo;re quicker to notice a low battery than to appreciate all other aspects of convenience they bring to our lives. Once something remarkable becomes routine, we stop noticing it at all.<br>I think we&rsquo;ve fallen into the same pattern with AI. The “strawberry” meme is a good example: we made fun of language models when they confidently miscount the number of r &rsquo;s in the word strawberry. It became a simple, memorable way to argue that AI wasn&rsquo;t really intelligent. At the same time, those same models could explain advanced mathematics, summarize research papers, write working software, and help millions of people learn new skills. Somehow the mistake became more remarkable than the capabilities.<br>But the joke is on us. We normalize competence almost instantly and grow increasingly sensitive to failure. Once something consistently exceeds our expectations, we stop rewarding it for doing so. We simply raise the bar and wait for the next mistake.<br>Intelligence Isn&rsquo;t the Impressive Part<br>After a year of working with coding agents, I&rsquo;ve come to appreciate that their most interesting quality isn&rsquo;t raw intelligence. It&rsquo;s their willingness to iterate.<br>They produce code that won&rsquo;t compile, misread requirements, reach for the wrong approach. Then they read the error, revise, rerun the tests, and try again. Then again. Sometimes dozens of times before creating something of value. Watching the process has made me realize that progress often depends less on getting the first answer right than on continuously incorporating feedback.<br>Humans aren&rsquo;t always as good at this. Once we&rsquo;ve invested time in an idea, it becomes difficult to abandon it, and our pride unconsciously folds hypotheses into identities. We defend decisions because they are ours rather than because they are correct.<br>Machines don&rsquo;t have that problem. They don&rsquo;t care whether yesterday&rsquo;s idea survives the afternoon; if the feedback says the approach isn&rsquo;t working, they move on. There&rsquo;s no ego to protect, and from the outside, it looks a lot like intellectual humility.<br>The Importance of Having an Objective<br>Working with coding agents has also changed how I think about intelligence itself.<br>I&rsquo;ve found that their performance depends surprisingly little on clever prompting and surprisingly much on the quality of the objective I give them. If I clearly define the problem, explain the important tradeoffs, enumerate the edge cases, and provide a reliable test harness, they can often complete ninety percent of the implementation with very little intervention. In those situations the bottleneck usually isn&rsquo;t the model. It&rsquo;s my ability to specify what success actually looks like.<br>The remaining ten percent is almost never about correctness. It&rsquo;s about judgment. Should this abstraction live here or somewhere else? Should the implementation optimize for readability or performance? Is documentation more valuable than self-explanatory code? Where should the single source of truth live? None of these questions have objectively correct answers. They are decisions shaped by experience, context, and instinct.<br>That distinction has become one of the most important lessons AI has taught me. Coding agents thrive because software engineering provides unusually clear feedback: every compilation, test suite, benchmark, and code review tells the system whether it&rsquo;s moving closer to or farther from the objective, and the objective itself is rarely in dispute. I don&rsquo;t think this is unique to programming, either. Mathematics, engineering, and much of science share the same property. Success is measurable, and reality answers with a clear “yes” or “no.” Those are exactly the environments where optimization shines.<br>But Life Doesn&rsquo;t Work That Way<br>Then I think about the questions that bother us and have a long-term impact on our lives. Whether to forgive someone who never apologized. Whether to leave a good job for one that might matter more. When, or whether, to have children. None of these are hard for lack of information. They are hard because there isn&rsquo;t a universally agreed definition of success. There is no compiler error for purpose, no benchmark for wisdom, and no test suite that tells us whether we&rsquo;re becoming the person we hoped to be. Instead, we navigate through experience, relationships, culture, memory, and intuition. Two thoughtful people can...