Evaluating your Agentic Harnesses | Data For Science

Subscribe to get our latest content by email.

Subscribe<br>We won't send you spam. Unsubscribe at any time.<br>Built with Kit

Companion code on GitHub<br>Notebooks & source for this series — DataForScience/LLMs

→ In previous posts, we borrowed Col. John Boyd’s OODA loop, built a “figher jet” harness around an LLM and even scaled from a single pilot to an air campaign with mission planners, squadrons flying independent sorties in parallel, fuel budgets, and a flight recorder. Now the pilot has returned to base and is time for a debrief.

No Air Force declares an aircraft combat-ready without it going through operational test and evaluation: flying against representative missions (including missions designed to break it) record everything, and compare the numbers against the requirements. Production level agents deserve nothing less, but almost never get it. Successful demos prove nothing more than the harness can work, but it takes an eval suite to demostrate that it usually works while quantifying the consequences of when it doesn’t.

We moved the entire harness (including typed tools, the plan DAG, the parallel executor, multi-tier memory, the verification hierarchy, multi-dimensional budgets, structured tracing, and the Orchestrator ) into a standalone module, d4sci_harness.py so that you can import it instead of rebuilding it:

import d4sci_harness as dh<br>from d4sci_harness import TOOLS, MemoryStore, Orchestrator

llm = dh.set_provider("anthropic") # or "mock" for offline runs

store = MemoryStore()<br>orch = Orchestrator(provider=llm, tools=TOOLS, memory=store)<br>The harness supports both a fully offline and deterministic mock backend, perfect for CI, and and an anthropic backend that gives you realistic planner, summarizer, and critic behavior. Switching backends is a one-line change. The rest of this post is based on live Claude runs.

Bare bones eval harness

Even a minimalistic eval unit should track a few useful metrics:

Pass rate: Functional correctness across tasks

Token usage: Cost predictability

Status codes: Failure mode distribution (failed_execute vs failed_verify vs budget)

Replan count : Robustness to bad inputs

The suite itself is a mix of correct and adversarial tasks aimed at exploring all code paths. In this example, we use three well-posed requests, and one that we expect to fail so that we can measure the recovery path:

EVAL_SUITE = [<br>{"goal": "Build a comparison report of Paris, Tokyo, and New York.",<br>"required_cities": ["paris", "tokyo", "new york"]},<br>{"goal": "Build a comparison report of London and Sydney.",<br>"required_cities": ["london", "sydney"]},<br>{"goal": "Build a comparison report of Paris and London.",<br>"required_cities": ["paris", "london"]},

# Atlantis is not in CITY_FACTS — this task exercises the<br># re-planning path, so we only require the satisfiable city<br>{"goal": "Build a comparison report of Paris and Atlantis.",<br>"required_cities": ["paris"]},<br>We loop through each task and extract the metrics out of the RunResult object our orchestrator returns:

async def run_eval(orch, suite):<br>rows, runs = [], []<br>for task in suite:<br>res = await orch.run(task["goal"], task["required_cities"])<br>runs.append(res)<br>rows.append({<br>"passed": bool(res.verdict and res.verdict.passed),<br>"status": res.status,<br>"replans": res.replan_count,<br>"tokens": res.budget.tokens_used,<br>"cost": res.budget.cost_usd,<br>"tier": res.verdict.tier if res.verdict else "n/a",<br>})<br>return rows, runs<br>And visualize the results in a simple table:

An overall pass rate of 75% and an average of 1,088 tokens and $0.0033 per task. Doesn’t seem too bad. We expect a pass rate below 100% as one task is expected to fail, and update the****status column with the cause of the failure (failed_verify vs failed_execute). Different problems require different solutions. The ill fated Atlantis task triggered one replan, as expected, confirming the recovery path was triggered correctly.

Paris is the only required city in the Atlantis task and is present in the report but the LLM judge , the last rung of the verification hierarchy, refused to call a one-city document a “comparison report of Paris and Atlantis”. This is the kind of nuance that LLMs excel at and that we catch because the eval records the tier that produced the verdict.

This suite is intentionally minimal but we architectured in a way that makes it flexible enough to be easily expanded.

A closer look at the flight recorder

The RunResult objects that the eval loop collects contain all the information we need, making it easy to provide useful visualizations.

We start by looking at the API costs. The Atlantis run is the most expensive of the four. It requires two rounds of planning and every tool call it makes along the way. Failure has a price tag!

Next, we look at the run time of the various tasks. The stacked segments sum the total work each run performed. The diamond marks actual wall-clock time.

The gap between the bars and the...