The AI industry's boldest promise right now is that models will soon improve themselves. Anthropic published a blog post titled "When AI Builds Itself" in June 2026. OpenAI advertised in July that its new model GPT-5.6 Sol had helped post-train a smaller model, saving researchers weeks of work. The implied timeline is that autonomous AI research is months away, not years.
A new study from Princeton pushes back. Researchers led by Peter Kirgis and Sayash Kapoor ran a method they call shadow evaluation: give a frontier AI agent the central research question from a high-quality unpublished paper, let it work for six days with $3,000 in compute, and have the original authors grade the output. The agents completed every engineering step without human help. Both papers were rejected, scoring 2/6 and 1/6 overall. The finding lands as the first controlled test of whether agents can do open-ended AI research, and the answer is a clear no.
What is shadow evaluation and how does it work?
Shadow evaluation sits between two existing ways of testing AI research agents. The first gives agents narrow tasks with checkable answers, like post-training a small model against a benchmark. That tests engineering but misses research judgment. The second submits AI-generated papers to blind peer review, which the study's authors call overstretched and noisy, since reviewers have limited time and variable expertise.
The new method works like this. The researchers picked two papers submitted to NeurIPS 2026 that had not been published. They gave Anthropic's Claude Opus 4.8, running on the open-source scaffold OpenClaw, the central research question from each paper. One question asked whether an LLM's personas can be controlled by editing model weights. The other asked how to design a detector that flags when a spreadsheet-prediction model becomes unreliable. Because the papers were unpublished, the agents could not memorize answers from training data.
Each agent got six days, $3,000 in Anthropic API credits, a GPU budget, its own virtual computer, and open-web access. The task: produce a research paper worthy of a top-tier AI conference. The original authors then graded the output using standard conference review criteria.
The agents reviewed the literature, ran hundreds of experiments, debugged GPU environments, retrieved external reviews via the web and email, and compiled full LaTeX documents. The only human interventions were logistical: solving scaffold issues, providing credentials, setting up the repository. On the engineering side, the agents were genuinely impressive.
How badly did the agents fail?
Both papers were rejected with high confidence. Paper 1, on persona control, scored 2/6 overall. Paper 2, on prediction reliability, scored 1/6. The reviewers expressed confidence levels of 4/5 and 5/5 respectively.
The chart above breaks down the scores by criterion, normalized to the maximum possible for each. Quality scores were 2/4 and 1/4, with reviewers citing unprincipled data and experiment choices. Clarity was 1/4 and 2/4: the papers were dense and hard to parse. Originality was the relative bright spot at 3/4 and 2/4, because the agents did propose some new methods, though built primarily on prior work. David Africa, one of the original authors, wrote that "the experiments and methodological choices were bizarre, and hard to understand." Viet Nguyen flagged a logical leap in the other paper: going from a few unsuccessful tests to the conclusion that no usable signals exist was "a kind of 'proof by example' fallacy that is highly non-scientific."
The agents also failed to meet basic formatting requirements. They ignored explicit instructions about time allocation, review frequency, and paper length. Both final papers would have been desk-rejected at a real conference for violating technical requirements alone.
Why can agents do the engineering but not the research?
The researchers identified five recurring failure modes that explain the gap:
- Poor judgment about the publishable bar. Agents understood the research questions and proposed hypotheses that resembled the original authors' early thinking. But they tested those hypotheses on tiny synthetic datasets, then declared them falsified.
- Uncreative responses to design flaws. When AI review tools and subagents flagged problems, the agents did not redesign experiments or reframe questions. They added caveats and narrowed their claims, producing papers with "extremely thorough negative findings rather than papers with new ideas."
- Ineffective backtracking. Both agents retired their most ambitious hypotheses within the first ten hours. One agent planned to spend 36 to 48 hours exploring three methods but stopped after five hours, latching onto the first method that showed any positive signal.
- Poor resource awareness. The agents did not manage tokens, compute, and time effectively, burning through resources without strategic allocation.
- Instruction drift. The agents ignored explicit rules about exploration time, review frequency, and paper length, producing work that violated conference submission requirements.
The pattern is consistent: agents started with their best ideas, ran underpowered experiments on those ideas, rejected them prematurely, and settled on progressively weaker approaches. The researchers noted that the hypotheses "grew narrower and less interesting" as the process went on. Even with a 48-hour exploration gate designed to slow them down, both agents still committed too early. The gate could be overruled by a subagent, and the agents overruled it.
The root cause, Kapoor suggests, is training. Models get good at what reinforcement learning can drill them on, and that works best when success can be checked automatically. Writing code, running benchmarks, debugging environments: these have clear pass/fail signals. Choosing a promising hypothesis, knowing when to abandon a line of work, deciding what evidence would actually settle a question: these do not. The same paper notes that "it is harder to create environments to train these models when the task itself is open-ended."
For all their failures, the agents did not engage in what researchers call reward hacking. They did not hide or misrepresent experiments. Subagents occasionally hallucinated results, but the orchestrator agent caught those errors. On integrity, the agents passed. On judgment, they failed.
What does this mean for builders betting on autonomous AI research?
If you are a founder or team lead planning a roadmap around AI agents that can do ML research, this study says the timeline is longer than the marketing suggests. Here is what the evidence supports right now:
- Agents can replace research engineers, not researchers. The agents debugged GPU environments, ran hundreds of experiments, compiled LaTeX, and managed compute, all without human help. If your team spends time on pipeline plumbing, agents are ready for that work today. If you expect an agent to find the next transformer architecture, you are betting against the evidence.
- Self-review loops do not fix bad research. The agents had access to AI review tools that surfaced many of the same issues the human reviewers later found. The agents did not act on the feedback productively. Adding more review steps to an agent pipeline does not solve the core judgment deficit.
- Instruction following degrades on long, multi-phase tasks. The agents ignored rules about exploration time and paper length, echoing what we have seen in our coverage of instruction collapse past six constraints. If you are building agentic systems for multi-day workflows, expect this failure and design guardrails around it.
- Premature commitment is the dominant failure pattern. The agents did not fail because they could not run experiments. They failed because they abandoned their best ideas after five hours of underpowered testing. Any agentic research system needs a mechanism that forces sustained exploration before commitment, and the study shows that a simple time gate is not enough.
This connects to the broader finding from our earlier reporting on how shadow evaluation exposes the gap between engineering and research. The engineering bottleneck is closing. The judgment bottleneck is not.
Is recursive self-improvement still possible without open-ended research?
This is the question the study raises but cannot answer. If recursive self-improvement only requires agents to grind on narrow, checkable tasks, like making training run faster or boosting benchmark scores, then the current trajectory might still get there. If it requires the kind of creative leaps that produced transformers or new architectures, the gap is real and the timeline is longer.
Kapoor calls this "the trillion-dollar question." Najoung Kim, a professor at Boston University who was not involved in the study, told MIT Technology Review that AI progress may bifurcate: fast on narrow tasks, slow on open-ended research. She also noted that concentrated investment could close the gap, even if agents are failing currently.
Anthropic's own Jack Clark offered a reading that rhymes with the findings. In his newsletter Import AI, he wrote that AI systems have "a certain absence of valuable, intuitive creativity" and called their rote, formulaic thinking a "bearish signal on short recursive self-improvement timelines." That is a striking admission from a company publicly betting on self-improving AI.
The study has clear limitations. It covers only two papers. The reviewers were not blinded: they knew they were grading AI-generated work, which could bias them toward harsher scores. And the researchers had substantial discretion in designing the study, meaning their own beliefs about agent limitations could have shaped the setup. The authors acknowledge that "evaluations of open-ended research trade some objectivity for a much richer test than any benchmarks can offer." Nature's coverage notes the same caveats while calling the approach a promising step toward holding AI to a higher standard than peer review.
The team is now running the same experiment with Anthropic's Mythos model, released in April and restricted to approved organizations under safety requirements. That result will be worth watching. If a more capable model fails the same way, the judgment gap narrows slowly. If it passes, the bottleneck may have been the model, not the task.
The trillion-dollar question
The study does not prove recursive self-improvement is impossible. It proves that the specific ingredient many forecasts hand-wave over, the ability to do open-ended research with judgment and creativity, is missing from today's frontier agents. Engineering capacity alone does not close the loop. Until agents can choose the right question, design the experiment that would actually settle it, and know when to throw everything out and start over, the self-improvement flywheel stays in manual mode. Anyone building a roadmap that assumes otherwise is reading the marketing and skipping the evidence.
Sources
- arxiv.org Kirgis, Kapoor et al., shadow evaluation paper
- nature.com Nature coverage of the study
- technologyreview.com MIT Technology Review article
- techxplore.com TechXplore coverage
- the-decoder.com The Decoder coverage
- aiweekly.co AI Weekly alert
