Building an Advanced Agentic Harness - by Bruno Gonçalves

SubscribeSign in

LLMs<br>Building an Advanced Agentic Harness<br>From a single pilot to an air campaign: planning, parallelism, memory, verification, and observability for production-shaped agents.

Bruno Gonçalves<br>Jul 15, 2026

Share

In Building a Basic Agentic Harness we borrowed Colonel John Boyd’s OODA loop and built the simplest thing that deserves to be called an agent: a loop that observes its state, asks the LLM to decide, validates the decision, executes a tool, and folds the result back into the state. The LLM is the pilot, and we built the fighter jet around it.<br>In this post we continue exploring the finer points of Agentic Harnesses and build a full fledge Advanced Harness.

Subscribe

The companion notebook is available on the LLMs for Data Science GitHub repository:<br>LLMs for Science GitHub

That Basic Harness loop is correct, but naive. A lone pilot in a well-built jet might win a dogfight, but nobody runs an air campaign that way. Real operations add mission planners who decide what sorties to fly before anyone takes off, squadrons that fly independent sorties in parallel, fuel budgets and bingo calls that force a return to base before the tanks run dry, flight recorders that make every mission reconstructible after the fact, and after-action reviews that decide whether the mission actually succeeded. None of these replace the pilot. They wrap the pilot in structure so that the whole system stays fast, safe, debuggable, and measurable.<br>Claude Code, Devin, Cursor, Hermes, and other production agents do exactly the same thing to the basic loop. In this post we upgrade every piece of our basic harness toward that production shape, without hiding any of the mechanics behind a framework. The guiding question for the whole exercise is a simple one:<br>How do you turn a single LLM call into a reliable system that can plan, act, recover, and prove it did the right thing?

Our answer is composition. We build small, testable primitives: typed tools, a plan DAG, tiered memory, a verification hierarchy, budgets, and a tracer, and wire them together with a deliberately thin orchestrator. Each primitive exists because naive agents fail in a specific, predictable way. LLMs invent invalid tool arguments, so we add typed tools with Pydantic validation. Everything runs sequentially, so we add a dependency graph and parallel execution. The context window fills with junk, so we add multi-tier memory under a retrieval budget. Bad outputs propagate silently, so we add a verification hierarchy. One prompt tries to do everything, so we split it into Planner , Worker , and Critic roles. Costs run away, so we add multi-dimensional budgeting with graceful degradation.<br>Proving the harness usually works, with an eval suite, retrieval benchmarks, and specialized worker pools will get a full fledge treatment in a future post.<br>The running example

Throughout the post we build a city comparison agent: given a list of cities, it produces a report comparing them on population, timezone, and a short narrative summary of each. The task looks almost insultingly simple, but it was chosen carefully. Each city-attribute lookup is independent of every other one, which means a three-city request naturally decomposes into nine tool calls that could all run at the same time. The final report, on the other hand, depends on all of the lookups finishing first, so we’re. well beyond a flat list of steps. We can programmatically check that every requested city actually appears in the report to verify the results. And the tools have wildly different costs: population and timezone lookups are in-memory dictionary reads, while the per-city summaries and the final aggregation each call the LLM, which gives us realistic budget pressure to manage.<br>For the sake of reproducibility, lookup tools read from a small mocked dictionary, CITY_FACTS , so the notebook is fully reproducible without network access. The LLM-backed pieces can run against either a real Anthropic model or a deterministic mock, which brings us to the first primitive.<br>A pluggable brain

Every component we are about to build eventually calls an LLM: the planner, the summarizer, the aggregator, the critic. If that call is hard-wired to one SDK, the entire harness becomes untestable and vendor-locked.<br>So before anything else, we define a base class that provides an abstraction over the details of the various LLM calling APIs<br>class LLMProvider:<br>"""Shared interface. Subclass to plug in a different backend."""

def complete(self, system: str, user: str, role: str = “default”) -> str:<br>raise NotImplementedError

async def acomplete(self, system: str, user: str, role: str = “default”) -> str:<br># Wrap sync call in a thread; works for any SDK.<br>return await asyncio.to_thread(self.complete, system, user, role)

We also implement a MockProvider for testing and debugging purposes that returns deterministic, role-aware responses: a canonical plan...