Can AI agents conduct open-ended AI research? Early evidence from two case studies | CRUXMenu

CRUX Evaluation 2<br>Can AI agents conduct open-ended AI research? Early evidence from two case studies<br>We gave frontier agents research questions from two non-public NeurIPS submissions and had the original authors grade the results. The papers produced by the agents were unambiguously rejected.<br>By Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, and Arvind NarayananJuly 30, 2026<br>Read the paper (PDF)·<br>Code, data, and agent logs

div>*:first-child]:mt-0">Abstract<br>Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.<br>Introduction<br>One of the most consequential open questions about AI capabilities is whether AI agents can conduct AI research. Many forecasts of explosive AI progress speculate that AI systems will soon do AI research themselves. This is also the explicit premise of leading AI labs; in June, Anthropic published a post entitled “When AI Builds Itself,” and in July, OpenAI advertised that their new model, GPT-5.6 Sol, had helped post-train a smaller model, saving researchers multiple weeks.11. Notably, GPT-5.6 Sol’s contribution to Luna’s post-training is not mentioned in the 81-page system card. But despite the significance of this research direction and the attention paid to these claims, the evidence base on whether agents can solve open-ended research questions is thin.<br>What unifies most of this recent work on autonomous AI research is its focus on verifiable tasks where agents need to improve a fixed, narrow metric. Benchmarks ask agents to improve a known metric, and an automatic verifier scores the result. Beyond benchmarks, a number of evaluations have shown AI agents beating expert human performance in tasks such as optimizing GPT-2 level models, using weaker models to train stronger ones, and optimizing an autoresearch harness.22. Appendix 4 describes these and many other AI research experiments that comprise verifiable tasks.<br>But much AI research goes beyond solving verifiable tasks. Agents can’t hill-climb their way into choosing a set of candidate hypotheses, deciding what evidence would settle a research question, or incorporating feedback effectively and recognizing that an approach has failed and the right move is to start over.<br>A small number of projects have adopted a different method for evaluating AI research: they submit AI-generated papers to blind peer-review processes, such as to AI conferences and workshops. But peer review is a weak measure of research ability: conference reviewing is overstretched and highly stochastic, and it does not reveal how many AI-generated submissions were rejected before the eventual acceptance.<br>CRUX is our project to conduct open-ended, long-horizon evaluations of frontier AI systems on challenging real-world tasks. It pushes frontier AI systems farther than benchmarks can by focusing on a small set of realistic challenges, on which we analyze AI systems’ performance deeply. Three months ago, we wrote a paper laying out the foundations of long-horizon open-world evaluations. In this evaluation, we ask: can agents solve open-ended AI research questions?<br>In this paper, we evaluate...