Can Claude Code in a loop improve a production enterprise AI agent with $10,745 of budget?
Jeremy Tian
SubscribeSign in
Can Claude Code in a loop improve a production enterprise AI agent with $10,745 of budget?
Jeremy Tian<br>Aug 11, 2026
Share
Over the past few months, agent harness optimization has become quite mainstream. We’ve been researching agent harness optimization for the past few years and have often been asked:<br>“Why can’t you just put Claude Code in a loop to improve your agent?”<br>It’s a really enticing idea, but we had some reservations, so we benchmarked it on a production-deployed enterprise AI agent (before we optimized it). We found some other popular optimization methods, and we pit them all against each other:<br>Claude Code in a loop
Autoresearch (Andrej Karpathy’s side project)
Autoagent (Kevin Gu at ThirdLayer’s open source agent auto-improvement)
The metric they optimized for was precision with a recall floor. They all started with the same agent configuration and were optimized on a dataset with the same judge. We ended up using a total of ~$67k exploring this experiment (including agent runs on the dataset, judge costs, etc.), and here are the results:
The baseline precision was 0.734. All three optimizers comfortably beat it.<br>Claude Code reached 0.818.<br>AutoResearch, 0.843.<br>AutoAgent, 0.877.
An interesting observation we made was that even though the 3 optimizers were given tens of thousands of dollars in compute, they found the best solution very early on.<br>The data suggests that these auto-optimizers can pick up low-hanging fruit, but cap out quite quickly. You can also waste a lot of compute time and money if you just let them run, hoping they find an agent configuration that breaks through the ceiling.<br>To improve an agent consistently, you can’t brute force it with more compute and budget. There are many more optimizations to be made, such as strategically running tests (most expensive part), escaping local minima, deciding which candidates to continue, etc.
Note: Claude Code was started later than the other two optimizers and was also slower. We also tested GEPA, but its results were very overfit.<br>*A lot of credits were hurt in this experiment
Thanks for reading! Subscribe for free to receive new posts and support my work.
Subscribe
Share
Discussion about this post<br>CommentsRestacks
TopLatest
No posts
Ready for more?
Subscribe
© 2026 Jeremy Tian · Privacy ∙ Terms ∙ Collection notice<br>Start your SubstackGet the app<br>Substack is the home for great culture
This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts