Guanghan Ning (@quietnning): "Opus 5 reports 30% on ARC-AGI-3, ~4× the previous best model, ~20× its predecessor Opus 4.8. We tested it on Witness, our held-out suite of ARC-AGI-3-style interactive puzzle games. The leap doesn't transfer.

On Witness composites (same harness, same budget for every model), Opus 5 lands at 43.4 ± 3.2, which is a statistical tie with kimi-k3 (42.8 ± 1.9) and Fable-5 (43.8 ± 9.7). Ahead of Opus 4.8 (34.8), but nowhere near a generational jump.

The traces tell the why:<br>(1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre.<br>(2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one.

That decomposition (perfect on templates, regressed on novelty) is the signature of “scaffold-then-internalize” training on genre-specific data, not a general gain in interactive abstract reasoning. Our benchmark can't tell whether that data was their in-house ARC-AGI-3-like corpus with ARC-AGI-3-specialized harnessing (likely thanks to [schema]? https://x.com/HavenFeng/status/2077770348876247502), or public Witness-genre corpus, or both, but it can tell the improvement isn't general.

Held-out evals only stay held-out while nobody's optimizing for the genre, and that clock is always ticking. It's ticking for Witness too, the moment we publish it." | XCancel

Guanghan Ning

@quietnning

16h

Opus 5 reports 30% on ARC-AGI-3, ~4× the previous best model, ~20× its predecessor Opus 4.8. We tested it on Witness, our held-out suite of ARC-AGI-3-style interactive puzzle games. The leap doesn't transfer.

On Witness composites (same harness, same budget for every model), Opus 5 lands at 43.4 ± 3.2, which is a statistical tie with kimi-k3 (42.8 ± 1.9) and Fable-5 (43.8 ± 9.7). Ahead of Opus 4.8 (34.8), but nowhere near a generational jump.

The traces tell the why:<br>(1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre.<br>(2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one.

That decomposition (perfect on templates, regressed on novelty) is the signature of “scaffold-then-internalize” training on genre-specific data, not a general gain in interactive abstract reasoning. Our benchmark can't tell whether that data was their in-house ARC-AGI-3-like corpus with ARC-AGI-3-specialized harnessing (likely thanks to [schema]? xcancel.com/HavenFeng/status/20777…), or public Witness-genre corpus, or both, but it can tell the improvement isn't general.

Held-out evals only stay held-out while nobody's optimizing for the genre, and that clock is always ticking. It's ticking for Witness too, the moment we publish it.

Claude

@claudeai

22h

Replying to @claudeai

On ARC-AGI-3, an evaluation where AI models must solve novel problems, Opus 5’s score is three times as high as the next best model.

Jul 24, 2026 · 10:46 PM UTC

15

50

330

32,389

Sort replies:<br>Relevant<br>Recent<br>Liked

Simon Ouellette

@SimonOuellette6

1h

Replying to @quietnning

This is why the public leaderboard should always be taken with a grain of salt -- as initially intended (somehow that goy lost along the way). The real research is what goes on in the Kaggle competition side of things.

258

Allegrant@BrunoOl10564268

23m

Replying to @quietnning

I'd wait to see Kimi-3's score in ARC-AGI-3, though, which I suspect will be relatively low.

That would indicate your benchmark and ARC might feel similar to humans, but are testing for very different things in AI models.

88

Greg Kamradt

@GregKamradt

2m

Replying to @quietnning

Thanks for sharing. First I'm hearing about this dataset, cool! Is it public to check out?

Couple notes/questions:

> On Witness composites (same harness, same budget for every model)

Are the # actions the same per game? If not, how does an even budget allocation work with games of varying lengths?

> (1) On our most classic Witness-style game

I read this as Opus 5 already knew the game. The game didn't test adaptation to novelty. Is that the case? What is the game? Is it similar to an existing game? What do you mean by "most classic"?

> (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8

What was the regression? What were the two scores on this particular game. fwiw we saw the same thing for our games. Some of them Opus 4.8 did better on.

> not a general gain in interactive abstract reasoning<br>>...