01: On Artificial Analysis | It's Monkey Business

← Index of /

Artificial Analysis has become a default leaderboard for frontier AI. When a lab releases a model, it is one of the first places people look instead of taking the vendor's launch slides at face value. That reputation is deserved in several areas: the throughput testing is useful, the pricing tracker is excellent, and there is real value in running hundreds of models through the same harness.

The trouble starts when all of that gets compressed into one score that is supposed to represent "intelligence."

The Intelligence Index is not a natural measurement. It is a collection of choices about what should count as intelligence, what deserves more weight, what gets counted twice, which old benchmarks stay after saturation, and how much credit belongs to the model versus the software wrapped around it.

I pulled apart the results from 586 model evaluations to see what those choices actually do to the index.

INDEX RECIPE / WHAT YOU'RE ACTUALLY SCORING<br>AA v4.1.1

Index Recipe

v4.1.1 Allocation

What the construct actually looks like<br>100% Total

50%

24%

26%

Agentic Loops (50%)<br>Coding (24%)<br>Knowledge (26%)

GDPval-AA v2 (Professional Deliverables)20%

Terminal-Bench v2.1 (CLI Systems Agent)16%

τ³-Banking (Multi-Turn Tool Agent)14%

Humanity's Last Exam (Text Hard Reasoning)12%

AA-Omniscience Accuracy (Fact Recall)8%

SciCode (Scientific Code Synthesis)8%

GPQA Diamond (PhD Science MCQs)6%

AA-LCR (Long-Context Reasoning)6%

CritPt (Physics & Math Frontier)6%

Omniscience Non-Hallucination4%

Published weights for the ten evaluations in Artificial Analysis Intelligence Index v4.1.1.

The 60-Second Summary

Version 4.1.1 moved the index much further toward "agentic workloads." The official Agents category rose to 34%, while General reasoning fell to 18%. That is a defensible choice, but it has consequences.

The short version is this:

GPQA is saturated, but it still gets 6%. Frontier models are bunched above 94% and separated by fractions of a point. It cannot reliably distinguish a 27B model from a 2.4T model.

The "24% Coding" category contains no repo-level SWE benchmark. Nothing in the index tests whether a model can enter a real multi-file codebase, trace a dependency chain, and fix a bug that spans several modules.

SciCode is closer to a science test written in Python. After controlling for general model capability, it correlates at r ≈ 0.71 with GPQA Diamond. It measures science knowledge more than software engineering.

GDPval gives the scaffold a lot of influence. When a fifth of the index depends on producing clean PDFs and spreadsheets, the Stirrup harness and its visual inspection loop become a large part of what is being evaluated.

Half the index uses the same basic loop. GDPval (20%), Terminal-Bench (16%), and τ³-Banking (14%) all rely on a recurring agent-execution pattern.

The weights barely change the ranking. Replacing AA's hand-tuned formula with a flat 10% split gives a 0.992 rank correlation and the same top ten models.

AT A GLANCE / CORE FINDINGS<br>586 CONFIGURATIONS

Statistical redundancy: PC1 accounts for 73.2% of total variance across all ten evaluations. The benchmarks mostly agree with each other.

Equal-Weight Rank Correlation (ρ)0.992

Top-10 Model Overlap (Equal vs AA)10 / 10

Agentic Execution Share50.0%

Dedicated Repo SWE Coverage0.0%

GPQA Frontier Dynamic Range (IQR)0.478

Summary metrics from a statistical audit of 586 benchmark evaluations across 409 underlying model families.

1. GPQA Is Still Here. Why?

GPQA Diamond used to be a very useful benchmark. At the frontier, it has been above 90% long enough that it is running out of room.

It contains 198 PhD-level questions across biology, chemistry, and physics. The design was elegant: subject-matter experts could solve them, while non-experts generally could not. AA runs the Diamond subset five times per model to reduce variance. That made sense two years ago, when the benchmark had more headroom.

Today the frontier leaderboard is pressed against the ceiling:

GPT-5.6 Sol: 94.1%

Gemini 3.1 Pro Preview: 94.1%

Claude Opus 5: 93.7%

The top ten models are separated by roughly two points. Comparing frontier models on GPQA right now is like timing Olympic sprinters with a stopwatch that only reads full seconds. When five models sit within 0.4 percentage points of each other, the differences you're seeing are stochastic sampling noise, not capability gaps.

I standardized every benchmark in the 586-model dataset to z-scores. GPQA has the lowest standard deviation, absolute range, and IQR (0.478) of anything in the index. It has the weakest discriminative power, yet it still gets 6%.

FRONTIER DISCRIMINATION / INTERQUARTILE RANGE<br>STANDARDIZED UNITS

94.1%

The Saturated Stopwatch

Frontier models compressed into a 0.4% margin. The ruler ran out of ticks a while ago.

CritPt (Math & Physics)1.832

Humanity's Last Exam...