Teaching the New Loop - by Andy Hall - Free Systems
Free Systems
SubscribeSign in
Teaching the New Loop<br>We must all learn to execute the new loop—to combine our human expertise with AI to produce private evals that measure how well AI is meeting our goals, then hill-climb against this measure.<br>Andy Hall<br>Jun 22, 2026
26
Share
“Without human direction, you have compute running in circles”<br>–Satya Nadella
Each frontier AI model release brings surprising new abilities that seem to shorten the list of what makes humans unique and wipe out the startups and established companies that thought they had differentiated themselves from this powerful new technology.<br>In a remarkable essay published the week before last, Satya Nadella took stock of this state of affairs and asked how any organization can thrive in a world where AI models continuously absorb the expertise of people and companies and sell it back as a commodity. This is not merely a question of business, Nadella tells us, but an existential question of political economy because it implicates the social contract and our shared trust that society can work for us all.<br>His answer to this conundrum is that firms have to combine their own human expertise with AI to codify their private knowledge inside model evals and training environments that they, and not the frontier labs, own. Nadella envisions a new “loop” where a company transforms its own workflows and accumulated judgment into systems that improve with every use, measuring the result against its own yardsticks rather than the public leaderboards, since, as he puts it,<br>“Private evals should capture whether a model is actually improving against outcomes that matter to the business.”
These private evals are what enable firms to remain sovereign because it allows them, and not the labs, to own their specialized knowledge. Nadella explains: “A company should be able to switch out a ‘generalist’ model without losing the ‘company veteran’ expertise built into their learning system. This is the key ‘test’ of your control and sovereignty in the era ahead.”<br>Nadella’s insights go far beyond the firm: indeed, the same opportunity exists throughout society. Think of citizens in a democracy, our political leaders, our universities…we all need a way to harness AI without being absorbed and captured by it. We should all be able to switch out one frontier model for another in our work, and in our lives, without losing our accumulated expertise. This new loop is the way.
So how should we all learn to build it? It’s not something covered in any standard curriculum. But I’m absolutely convinced that it needs to be.<br>I say this in part because I’ve been experimenting with it in my teaching this year. Just a few weeks before Nadella published his essay calling for companies to build private evals, I had my students in the new “Free Systems” class I’m teaching at Stanford GSB do exactly the same thing. With Claude Code subscriptions, OpenRouter API credits, and a couple of weeks of practice under their belts, my students all created first drafts of their own personal evals in a single three-hour class session.<br>Each student identified a criterion they cared about—how sycophantic the models were when debating controversial topics, how well the models gave voting advice, whether they understood crucial cultural nuances across languages, and much more—and then designed a rubric for scoring model responses according to their personal beliefs. Then, they built a leaderboard comparing how different leading AI models performed according to their measure. The results were spectacular! But we weren’t done.<br>Once they could measure models against their own personal yardsticks, we spent the rest of the quarter pushing further—what else could students do with this power? It turns out, a lot. Their final projects give a good sense of what the future that Nadella describes might look like. Together, they look like a nascent field guide to building these new loops in the wild.<br>What the students did
Here is what that future looked like in our class: fifteen final group projects spanning finance, governance, media, and security, several of them grown straight from the personal evals the students had built a few weeks earlier.<br>While many of the projects are, at their core, evals, they go beyond the evals we built in class because they’re not just leaderboards; instead, the evals are embedded into broader tools that do something based on the information—whether that’s make a new recommendation, helps the user complete an action, give better advice, and so on. This is how they begin to show us what the new loop will look like.<br>I’ve grouped them below by theme, with a great deal of help from Claude. You can see all of the projects at this website.<br>Finance and delegated decision-making<br>AI Bank Run Simulator (Shang Jing Chia) — live simulation of an AI-driven bank run with LLM agents acting from distinct personas; lets you watch cascades...