Automating AI model research with evolution - Imbue

Subscribe<br>Menu Article/Research<br>Automating AI model research with evolution

17 min read<br>Last updated 22 Jul 2026

Daniel MewesResearch

Key takeaways<br>Introduction<br>Nanochat auto optimization<br>Why linear agents get stuck<br>Escaping dead ends by evolving interpretations<br>The experiment efficiency vs. diversity tradeoff<br>What’s next?<br>Appendix: Reproduction details

Key takeaways<br>We’re open-sourcing Imbue Catalyst, an AI tool for computational research and scientific discovery<br>Our evolution-based optimizer improves nanochat LLM performance 3x further than a regular AutoResearch agent<br>Introduction<br>Recent model and harness improvements have enabled practical use cases of “self-improvement”: AI systems that iteratively improve their own foundations.<br>Approaches such as Darwin Godel Machine focus on improving a system’s harness, while others optimize the training recipes, architecture choices and algorithms behind the models themselves (Andrej Karpathy’s AutoResearch, Recursive’s “First Steps Toward Automated AI Research”). Even though many of these results have been on small “toy” models, it is becoming clear that the next generation of LLMs is already being built with the help of today’s AI systems (e.g. Anthropic “When AI builds itself”).<br>Today, we’re announcing Imbue Catalyst. Catalyst is a research tool that uses evolution-inspired methods to help with the following research tasks:<br>Optimize a piece of code, algorithm, or model with respect to a given metric<br>Find solutions that satisfy certain programmatic verification criteria<br>Discover explanations for computationally reproducible phenomena<br>Assist with reviewing, formalizing, and editing theories in computational research fields

We believe that AI research works best as an open, collaborative effort. Hence, we’re releasing Imbue Catalyst under an AGPL-3.0 license. We’re looking forward to seeing what you’ll discover with it!

In this post, we’ll use Catalyst to optimize nanochat, a small transformer language model trained from scratch, and dive into the mechanisms that allow Catalyst to continue discovering improvements well after regular coding agents level off.<br>Nanochat auto optimization<br>Andrej Karpathy first proposed the AutoResearch setup, in which a coding agent is prompted to improve the training setup for a small transformer language model (nanochat). It is asked to do so by iteratively running experiments, and using git to keep track of its progress.<br>The setup starts out with a training script that runs for exactly 5 minutes on a single H100 GPU. The agent is free to change the model architecture, training recipe and hyperparameters in order to minimize the resulting BPB (bits per byte) metric on a validation dataset, as long as it doesn’t exceed the 5-minute training time budget.<br>Nanochat training optimization is an example of what we call a verifiable goal in Catalyst. Verifiable in this context means that you can use a program to measure and/or verify how well a candidate solution achieves the goal at hand. In this case, the verifier is measuring the BPB metric of the resulting model.<br>We compare Imbue Catalyst’s verifiable goal solver, ”Catalyst (Evolution)”, against the following reference points:<br>Plain AutoResearch. We show the “mean of 3 runs” data reported by Weco AI.<br>Recursive Superintelligence’s recently reported results. With two caveats:Recursive switched to a different GPU (B200) part-way through their run. We only include the first part of their run that ran on the original H100 GPU.<br>Recursive did not report the number of experiments in their blog post, but rather reported cumulative wall-clock time. It is unclear from their post whether their system executed multiple experiments in parallel, and what the typical duration of each experiment was. Thus, our x-axis scaling of their results is estimated and might not correspond to the true experiment efficiency of their system.

For ablation purposes, we also compare against a simplified “linear” configuration of Catalyst’s solver. In this configuration, the evolution-based mechanisms are disabled. All other factors (base LLM, hardware, prompts, etc.) are kept identical to our main result to allow for an apples-to-apples comparison.

* Number of experiments (x-axis) is estimated for Recursive’s run

Catalyst’s evolution-based solver significantly outperforms both the original AutoResearch and Catalyst’s own linear ablation baseline.<br>In the measured run, Imbue Catalyst achieved a val_bpb score of 0.9361. Notably, when we stopped the run after 340 experiments, val_bpb had not plateaued yet, suggesting that further improvements could be possible given enough time and compute budget.<br>Our solution also surpassed the best result achieved by Recursive’s reported H100 run, but it is uncertain whether their solution would have continued improving if they had kept running on the H100 for longer. We would also like to re-emphasize that the number of...