Training Agent Harnesses Like Model Weights | Henry Pan
Project Repository: https://github.com/workofart/harness-training
So I recently wanted to see whether an AI agent could self-improve a harness to solve terminal bench tasks. To align on the definitions, “harness” means the system (e.g. Claude Code, Codex, ChatGPT web interface etc…) wrapping around the model (e.g. GPT-5.5, Claude Opus 4.7 etc…) that interacts with a specific environment. The harness controls what the model sees, what tools the model can use, and how environment responses are fed back to the model etc…
My initial attempt was mostly treating the self-improvement loop as an experiment design problem. I wrote about the learnings in my previous blog post What 1,000+ Harness Experiments Taught Me About Self-Improving Agents.
Since then I’ve completely revamped the way I run experiments (hint: determinism). This allowed me to reduce the experiment time, to extract more signal per experiment and to build a general PyTorch-like framework for “training harnesses”. In this blog post, I’m going to talk about how this is done and why it matters in the context of recursive self-improvement.
Here are some results from evaluating the same trained harness + model X:
Training the Harness
For a precise definition, this problem is framed as an agent-guided discrete program search with terminal reward and persistent search memory.
Here’s a map of the recurring concepts and the PyTorch analogy that inspired the framework (introduced later):
Concept<br>PyTorch/Machine Learning analogy
Harness (core.py): The program being trained. The current best version is the baseline; a proposed edit to the python file is a candidate<br>Model weights
Improvement agent (Codex CLI or Claude Code CLI): Reads the experiment evidence, proposes one bounded harness change per epoch (iteration) (AgenticEstimator in the framework)<br>Gradient estimator
Task LLM : Executes the training tasks through the harness; never changes during training<br>Frozen backbone
Criterion : Compares candidate vs. baseline task outcomes to determine whether the candidate was good<br>Loss function
Optimizer : Promotes the candidate to new baseline (git fast-forward) or rejects it (preserved as a git ref)<br>optimizer.step()
Experiment run : Full execution of the training task set under one candidate<br>Training epoch
Learning memo (learning.md): Persistent memory across epochs: what worked, what didn’t, what to try next<br>Optimizer state
Setup
My setup is pretty basic, mostly to control costs and reach a decent iteration speed of one experiment (30 - 40 tasks) within ~45 minutes:
Rented 5090 GPUs in the cloud (1 - 4 depending on how many concurrent tasks I’m running). The LLM inference providers don’t guarantee determinism, which I will explain later.
Qwen 3.6 35B A3B FP4 quantized model : The model weights occupy ~24 GB, while the KV cache (with radix/prefix caching disabled) uses ~4 GB to support up to 10 concurrent requests, a limit imposed by the deterministic kernel(discussed later). The remaining memory is consumed by activations and CUDA overhead.
LLM inference configurations: seed 12345, temp 1.0, top-p 0.95, 32k context length, max token count 8k, reasoning token count 6k. These were mostly constrained by GPU memory and aiming decoding time of below 5 minutes for the worst-case max token count, and I didn’t want to split the model across GPUs
SGLang inference engine: it supports deterministic batch inference, which I will explain later why it’s a prerequisite for training the harness (bootstrap script)
Next, I will dive into the two types of tasks where I trained my harness: SWE bench tasks and Terminal Bench tasks.
Baseline Harness
It’s important to mention how I define my baseline harness because this is kind of like how one would initialize a machine learning model during model training. And it is very important to initialize your harness properly because otherwise it’s going to be very hard for the harness training to find good learning signals (gradient). For example, I’ve seen that if the baseline harness is too thin, without the LLM response handling logic, most of the training will be spent on creating error handling for the LLM calls. The other extreme is when the baseline harness is overloaded with 10+ tool definitions, at that point there’s too much bias. Maybe certain models are trained with that, but it’s definitely not a generic starting point.
In my baseline harness, I defined two actions in total for tool use: Run for running any shell command and Submit for letting the harness know it’s ready to call the task verifier to check the results. By no means is this a good baseline, but it’s a clean yet learnable state.
Training on SWE Bench Tasks
Training Setup:
Trained on 39 SWE-bench tasks (django/pylint/pytest/sphinx/sympy) 1
Capped at max 110 steps per task or 75 mins timeout per task
Frozen Task LLM: Qwen 3.6 35B A3B FP4 quantized model
Agent used as...