Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT - METR
Our Work
Research
Notes
Updates
Risk Assessment
About
Donate
Careers
Search
-->
Our Work
Research
Notes
Updates
Risk Assessment
About
Donate
Careers
Menu
×
Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
CONTRIBUTORS
Tom Cunningham,
Manish Shetty,
Vincent Cheng,
and
Nate Rush
DATE
July 21, 2026
SHARE
Copy Link
Citation
BibTeX Citation<br>×
@misc{metr-2026-expenditure-horizon,<br>title = {Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT},<br>author = {Tom Cunningham, Manish Shetty, Vincent Cheng, Nate Rush},<br>howpublished = {\url{https://metr.org/blog/2026-07-21-expenditure-horizon/}},<br>year = {2026},<br>month = {07},
Copy
We propose a measure of an AI agent’s optimization ability with an “expenditure horizon.” We give an empirical illustration from the NanoGPT speedrun.
One difficulty in measuring AI’s ability to accelerate AI R&D is accounting for token cost, experiment compute cost and human labor cost. If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. We illustrate our method with data from the NanoGPT speedrun. We first estimate the returns to human effort in optimizing NanoGPT: on the margin, each 1% improvement costs very roughly \$2,500 in human labor. We also report the status of preliminary agentic optimization runs against NanoGPT: after more than \$10K of expenditure we estimate expenditure horizons of \$0-\$3K.
Introduction
A critical question is the degree to which AI will accelerate its own progress.
Over the last year there have been many credible claims that AI has begun accelerating algorithmic progress but it is hard to tell by how much, and what to expect in the future.1 We have many imperfect sources of evidence on AI’s acceleration of AI R&D:
AI R&D benchmarks. There are a few high-quality benchmarks which test agents on their ability to optimize AI training algorithms. However most do not report any human baseline, or only a baseline score at 8 hours or 40 hours. Additionally, the benchmark problems are often not reflective of frontier-level AI R&D. It seems plausible that agents can get good solutions to many “textbook” or “toy” problems without being able to contribute to highly-optimized frontier algorithms.
Researcher-uplift studies. Ideally we would compare researcher productivity with and without AI help, but it is very difficult to run an experiment that estimates the effect accurately (discussion in Becker et al., 2026). We do have some self-reported estimates of productivity improvement: around 4X from the Mythos Preview system card; around 2X from Becker/METR, but both reports recommend a great deal of caution in interpreting these claims. Recent studies find AI significantly increases lines-of-code produced (Anthropic reports that Mythos increased merged lines of code by 8X), but production of code is difficult to map to productivity in R&D.
Qualitative reflections. The Mythos Preview system card describes (1) survey results on ability (4/18 respondents thought Mythos Preview could replace an entry-level researcher with 3 months of scaffolding iteration); and (2) discussion of strengths and weaknesses of Mythos Preview in doing AI R&D. These are useful but again difficult to interpret as quantitative measures of acceleration.
Contributions to frontier optimization problems. There have been many recent reports of autonomous or AI-assisted contributions to frontier optimization problems (some of which are AI R&D problems) e.g. TTT-Discover, AlphaEvolve, and LLM-assisted NanoGPT speedrun contributions. However it is difficult to assess the magnitude of these contributions relative to human effort, and reporting is biased towards successes making the results hard to generalize.
Acceleration in capabilities progress. The evidence above was regarding AI’s contribution to AI R&D inputs; we can additionally look at the trajectory of output, e.g. looking at acceleration in capabilities as measured by METR’s time-horizon or Epoch’s ECI. Estimating how much progress is attributable to AI R&D requires also estimating the contribution of human R&D and training compute (e.g. as done in the Mythos Preview system card). This is intrinsically a lagging metric, only observed after the capabilities are realized in a model.
The method in this note is a novel metric for summarizing an agent’s ability on a optimization problem. It thus can be used to summarize performance on existing AI R&D benchmarks (e.g. RE-bench, MLE-bench), but we also show how to apply it to agentic contribution to certain frontier optimization problems, and we illustrate with NanoGPT.
An agent’s “expenditure horizon” gives a quantitative...