What Evolves When We Talk About Harness Evolution?

Profile<br>Blog

Section 1

Dissecting Gains from Harness Evolution

Figure 1 · Where each reported gain comes from

OOverfitting<br>TTest-Time Scaling<br>GGeneralizable Improvement

An agent system can be viewed as an LLM paired with a harness: the prompts, tools, memory, skills, orchestration code, and control flow that shape how a model interacts with a task.

Recent work[1], [2], [3] automates harness engineering through harness evolution, a form of text-space optimization that iteratively updates the non-parametric system around a fixed model using task feedback. In each iteration, an executor model attempts the tasks with the current harness. A proposer agent then inspects the trajectories and evaluation results to propose edits to the harness. Recent systems following this paradigm have reported intriguing gains from evolved harness code[1], skill files[4], and updated tool-use or orchestration patterns[5].

However, a higher aggregate score does not reveal what changed in the system or whether the gains will persist beyond the search set (training split). The same score increase can hide different mechanisms:

Overfitting<br>e.g., hardcode a recurring answer pattern from the search set.

Test-Time Scaling<br>e.g., sample five responses in parallel and take the majority answer.

Generalizable<br>Improvement<br>e.g., break tasks into reusable substeps or guard against common errors.

These mechanisms have different implications for cost and generalization , even when they produce the same benchmark gain.

We therefore separate harness-evolution gains into three mechanisms: Overfitting, Test-Time Scaling, and Generalizable Improvement.

We define Overfitting as gains that rely on patterns specific to the optimization setting and therefore do not persist under an appropriate distribution shift. This includes exploiting dataset artifacts, or distilling task-specific knowledge from the proposer model. For example, a harness may string-match a recurring answer option if it is always correct when it appears in the dataset. Similarly, on well-studied synthetic tasks, the proposer may write deterministic code tailored to the current setting without calling the executor LLM at all. Such gains raise the search-split score without improving how the executor handles new tasks. Among the three mechanisms, overfitting is therefore the most concerning and should be interpreted with the greatest caution.

Test-Time Scaling sits in the middle ground. It happens when the evolved harness spends more inference compute through, for example, retries, verification, or parallel sampling[6]. The resulting gains may be matched by simply scaling the baseline to the same compute budget. While gains bought by extra compute may transfer to held-out tasks, whether they are worthwhile depends on the use case and the sensitivity to cost.

We define Generalizable Improvement as gains that remain after accounting for overfitting and test-time scaling. These gains may come from reusable skills, better task decomposition, or methods that address common executor errors. They are most desirable because they are more likely to work on new tasks without using more compute.

Figure 1 previews our main and most striking finding: across representative benchmarks in math, coding, creativity, and agentic tasks, most search split gains are explained by Overfitting or Test-Time Scaling; and the residual generalizable improvement is often small. The transferability of evolved harnesses to held-out test sets is even more limited, as shown in the transfer analysis.

In the following sections, we will explain how we attribute the gains with Harness-Delta Attribution , directly evaluate the generalizability on held-out tests, and show that preventing overfitting remains non-trivial even with validation gating. We further discuss how harness evolution interacts with the executor model&rsquo;s base capabilities and other task-specific details in the appendix.

Section 2

Our Experiments

Harness-Delta Attribution: How Much Does Each Mechanism Contribute?

We introduce Harness-Delta Attribution (HDA) (method spec), a procedure that decomposes the score difference between a baseline and an evolved harness and quantifies the contributions of overfitting, test-time scaling, and generalizable improvement.

Figure 2 · Harness-Delta Attribution at a Glance

1. Compare the scores S(B) and S(E) to get the observed gain.

observed gain: S(E) − S(B)

T · Test-Time Scaling<br>G · Generalizable Improvement<br>O · Overfitting

T = S(Bcc) − S(B)<br>G = S(Eneutral) − S(Bcc)<br>O = S(E) − S(Eneutral)

base harness

Bcc<br>B compute-matched to E

Eneutral<br>E with overfitting neutralized

evolved harness

Let B denote the baseline harness and E the evolved harness. HDA evaluates two controlled variants: Bcc, which matches the baseline to the evolved harness's inference budget, and Eneutral, which neutralizes detected overfitting mechanisms in the evolved...