Most "self-improving" AI agents don't actually improve

SubscribeSign in

Most "self-improving" AI agents don't actually improve<br>Out of 9 running in production, only 2 had a key element: verifier grounded in the real world

Kamila Selig | AI @ Big Tech<br>Jul 22, 2026

Share

Exec summary<br>Despite the hype, public examples of true self-improving loops are few and far between.

A self-improving loop is only as good as its verifier, and a verifier is only as good as the real-world data grounding it.

Building verifiers is hard, slow, and complicated, and where most of the competitive opportunity is today. Most AI engineering is trying to side-step them.

Before you sponsor a project that builds one, align the resources: (1) what real-world data grounds the verifier? (2) who provides expert feedback on failures and what does that cost?

Subscribe

Loops are having a moment. The poor souls who take it upon themselves to be their organization’s Most Plugged-In AI Person were in for a treat over the last few weeks: Boris Cherny, (creator of Claude Code), shared his Steps of AI Adoption, a de facto blueprint of what it takes to move past the bottlenecks in automating work; co-founder of Arize AI Aparna Dhinakaran broke down the theory of loops. Gergely Orosz of The Pragmatic Engineer quickly followed with a piece rightfully dispelling some of the hype.<br>No doubt many organizations are experimenting with self-improvement. But if you look for documented cases of self-improving loops in production , the list gets short quickly; among 9 use cases I found and looked into, only two disclose enough to verifiably call the loop “self-improving”.<br>Verifier is key

In its simplest form, a loop has three components:<br>The execution loop, which is whatever the automation actually does; say, monitors the bug queue and fixes them;

The verifier - checks the output of the execution loop against some standard you established (if in your mind verifier is served well by golden-data-evals alone - please read along);

Feedback - if the verifier flags an issue with the execution loop, the context of the issue is fed back into the execution loops for another spin.

And it’s in the verifier the real bottleneck of “self-improving” lives.<br>Why verifiers are expensive and difficult

Verifier, despite what seems to be an increasing trend in the industry, must be grounded in a steady trickle of real-world data; if it isn’t, it can (and will) optimize in a random direction.<br>For deploying code, this “steady stream of real-world data” is relatively straightforward, even if it requires some creative stitching: tests can pass or fail, production telemetry can catch anomalies, code history can be audited, A/B tests can run, and user feedback and bugs queue can be monitored.<br>As you move further outside of code and into knowledge work, figuring out the “steady trickle of real-world data” needs some creativity.<br>This need for creativity is why I like to see how various businesses have tackled the problem, but evidence on in-production loops is pretty thin. Kudos to the folks behind the two examples I break down below: Replit’s product development agent and Tax AI agent, built by OpenAI’s FDEs (PS. are FDEs a revival of 2010s implementation consulting?).<br>Agent drafts, human ships: current state of self-improving loops

Exhibit 1. Replit’s self-improving coding agent<br>The use case for Replit’s agent is straightforward: identify product issues, write the code and submit the build. From here, verifier takes over, including Replit’s benchmark built specifically for vibe-coding: it simulates the app and user’s clicks within it; if passed, the proposed change runs through A/B testing, and results are sent to a human for review to ship, iterate, or drop.<br>For me, an even more interesting element of Replit’s loop is Telescope - their proprietary traces clustering tool1, which allows to analyze traces at scale without unnecessarily exposing user data to random eyeballs. Plus, it looks cool.

Replit’s self-improving agent. Green: verifier components, black: execution loop.<br>Exhibit 2. Tax AI agent, built by OpenAI x Thrive<br>Tax AI set out to tackle the ambiguous and spottily-documented process of tax preparation. Since wading through hundreds of documents to prepare a draft is the most labor-intensive part of the CPA’s job, the appetite for automation was high enough to dedicate the time to weeks of painful - but effective - mechanical turking.

What I delightfully noted in OpenAI’s use case is that the agent improved drastically post-launch2. The percentage of drafts that were at least 75% correct went from roughly a quarter at launch to 86% - vast majority - at six weeks. The result comes from FDEs pulling teeth asking CPAs to explain each decision where their judgment differed from the agent.<br>Think about this process: one tax draft has dozens - or hundreds - of fields, each requiring data and judgement - sometimes formal, sometimes not. Imagine those six weeks and the...