Tuning the harness, not the model: a Nemotron 3 Ultra playbook

Learn

DocsCompany

PricingTry LangSmith

Get a demo

Try LangSmith

Get a demo

Case Studies

Partner

Tuning the harness, not the model: a Nemotron 3 Ultra playbook

Nick Hollon

Srimanth Tangedipalli

July 8, 2026

11<br>min

Go back to blog

Create agents

Share

Key Takeaways<br>Near-frontier agent quality at a fraction of the cost. Tuning the harness alone took Nemotron 3 Ultra to a best run of 0.86 on the Deep Agents suite, nearly matching Opus 4.8's best of 0.87, at roughly 10x lower cost per run (about $4.48 against $43.48 on the full suite) with latency at parity.‍<br>Evals are the training data for harness work. Every change ran through a trace-driven loop, screened cheaply first, and earned its place only if the win repeated across trials and regressed nothing else.‍<br>Fit decides how much capability reaches the task. A matched harness lets the model spend its capability on the work; a mismatched one makes it fight the scaffolding, and the gap between the two shows up in the score without touching the weights.‍<br>Harness tuning has a ceiling. It fixes failures that come from the scaffolding, but it can't add what isn't in the weights, so a result that stays flat through every harness change points to post-training rather than another hook.

An agent is a model plus a harness. The model does the thinking, and the harness (the system prompt, the tool descriptions, the middleware) is the scaffolding it works inside. We've tuned harnesses around frontier models before, but, this time, we wanted to see how far we could get with an open model.<br>Open models are where this gets interesting. They've gotten good enough to take seriously for real agent work, and they cost a fraction of a frontier API. You get the weights, so you can host and fine-tune the model yourself, or you can use an endpoint from a variety of Cloud providers without lock-in. The catch is that a capable model can still underperform in a harness that wasn't built for it, which is the part we set out to fix.<br>As a member of the Nemotron Coalition, we thought Nemotron 3 Ultra was the right model to tune inside Deep Agents. NVIDIA built Nemotron to work inside agent harnesses, and we wanted to see how far we could take it.<br>The harness is the part you control<br>Out of the box, a generic harness is not tuned to the model. Using a model without harness tuning is a reasonable default but not best you can do.<br>The harness is everything around the model, and the model is the engine inside it. When the two are matched, the model spends its capability on the task. When they are not, it spends capability fighting the scaffolding, re-asking for details it already has, stopping early, or looping.<br>The fit matters more than most people expect, and we've shown it before. On Terminal-Bench 2.0, we took gpt-5.2-codex from 52.8 to 66.5, roughly Top 30 to Top 5 at the time, without touching the model. When we shipped per-model harness profiles, we improved a curated subset of tau2-bench by 10 to 20 points by conforming to prompting guides. The same weights with different scaffolding lead to a different score.<br>We did that harness-side work using a data-driven approach, mining traces for failure patterns. The case study is Nemotron 3 Ultra, an open model that already comes a long way on its own, because NVIDIA post-trained it specifically to behave consistently across agent harnesses, not just single-turn chat, on a large suite of long-running, tool-using tasks (NVIDIA's launch post covers the agentic post-training and the architecture behind it).<br>We kept the model fixed and changed only the harness: the system prompt, the tool descriptions, and the middleware around model and tool calls. Generation settings stayed at the vendor-recommended defaults, so nothing below comes from changing temperature, top-p, or thinking budgets.

Start with evals<br>Tuning a harness to a model always starts with evals. Without a learning signal you are guessing, and a harness tuned by guessing overfits to whatever you looked at last. We treat evals as the training data for harness engineering: each case contributes a signal about whether the agent took the right action or produced the right outcome, and that signal guides the next change.<br>Agent development does not look like normal software. With normal software you usually know the inputs, the outputs, and the expected behavior. With an agent the input space is wide, many outputs are acceptable, and a small change to the prompt, the tools, or the orchestration can fix one case and break another. So the work is iterative, run as a loop with the LangSmith trace as the source of truth:<br>Evaluate. Run the behavioral suite across the models.<br>Observe. Read each failing run's trace to see where and how it failed.<br>Diagnose. Cluster the failing trajectories into behavior patterns.<br>Engineer. Make one targeted change of a piece of the harness.<br>Re-evaluate. Promote the change up a cost ladder,...