FrontierCode 1.1 | Cognition<br>Menu

Close

One month ago, we introduced FrontierCode [1], an eval designed to measure not just code correctness, but also code quality. Today, we are releasing a refined version, FrontierCode 1.1 , with the following improvements:<br>Fair internet use. We refined our methodology to capture the nuance between legitimate internet use (e.g., looking up documentation) and unfair use (anything that could reveal a task's solution).

Fairer grading. We audited all 1000+ grading criteria and relaxed 75 overly strict ones that could unfairly penalize valid solutions.

New model scores. We are releasing scores for Sonnet 5 and updated scores for Fable 5.

Going forward, we'll be reporting scores on the Main and Extended subsets of FrontierCode, evaluated with the 1.1 methodology. We are no longer reporting the Diamond subset; we elaborate on this below.

Results#

We present the results on FrontierCode 1.1 Main above. While the FrontierCode 1.1 methodology leads to some changes in absolute scores, the relative performances of the models we evaluated did not substantially change compared to 1.0.<br>In the rest of this blog post, we detail our improved methodology for proper internet use.

Getting Internet Use Right#

FrontierCode tasks are sourced from real PRs in open-source codebases. This produces a realistic task distribution, but it also means that task solutions may exist somewhere on the public internet, such as in a later version of the upstream repository, in its many mirrors, or even in package registries (an agent can sometimes obtain the fix simply by installing the latest release). An agent with internet access can therefore sometimes shortcut a task by looking up the answer instead of solving it.<br>When designing FrontierCode 1.0, we found a few instances of agents finding the upstream codebase, but occurrences were rare enough that we felt they did not warrant an explicit correction or methodology change. However, the latest models such as Fable 5 are increasingly skilled at retrieving information online, and the rate of unfair internet use is rising accordingly. Without any instructions to the contrary, finding an existing fix is a natural strategy for a capable agent. We expect this trend to continue with future models.

Why not just turn the internet off?<br>Unfair internet use is an increasingly common issue in SWE evals [2,3,4,5], and the usual fix is to disable internet access entirely. But a blanket ban has two serious problems:<br>Several FrontierCode tasks require internet access by design, for example, to look up API contracts. This reflects real-world software engineering and was an explicit goal of the benchmark.

Even for tasks that don't require it, frontier models are increasingly trained to use search as a core part of their reasoning and context-gathering workflow. Disabling internet access removes this capability and can cause benchmarks to understate true model performance.

For these reasons, FrontierCode did not (and still does not) disallow internet use. Instead, FrontierCode 1.1 aims to eliminate unfair internet use while preserving the realism that internet access provides.<br>Our approach: define fair internet use, then verify<br>We found that the latest models are sufficiently well-aligned that simply telling them which kinds of internet access are allowed (e.g. looking up documentation) versus disallowed (anything that could shortcut the task) almost entirely eliminates unfair internet use, while still permitting fair internet use. Adherence to the prompt is remarkably good: with it in place, unfair internet use rates fall below 1% for every model we evaluated.<br>FrontierCode 1.1 implements this as two safeguards: a prompt that explains what fair internet use is, and a classical verifier that detects unfair internet use and zeroes out those runs. The prompt alone is currently sufficient to essentially eliminate unfair internet use; the verifier confirms this and will catch any future deviations.<br>Safeguard 1: a "fair internet use" prompt that clearly distinguishes allowed from disallowed internet use.<br>View the fair-internet-use promptShow<br>We provide two illustrative examples of models’ internet use below:

Flagged: opened the PR diffAllowed: read the docs<br>The model can search the web freely, but the scanner flags the run the moment it opens the upstream PR diff page.task: Ellipsis on long emails

0/14<br>Scroll into view to watch the agent work…

Interactive: watch the agent work in real time. Routine steps play fast; the scanner slows down on the moment the run is flagged, when the model opens the upstream PR diff page.Safeguard 2: programmatic detection. We flag references to source pull requests, upstream patches or files, and potentially solution-bearing mirrors or vendored copies. To penalize unfair internet use, flagged runs receive a score of zero.<br>Together, these safeguards eliminate essentially all unfair internet use while still allowing fair use. Agents...