Harnessing Harnesses - Climbing the LLM Hills
Search<br>Ctrl+K
Contents
In my previous LLM-themed post I talked about the use of MCPs and how they have been useful for my offensive bug hunting pipeline. MCPs grant Large Language Models(LLMs) access to additional tools and allow them to gather additional context. This post will explore the creation and importance of harnesses for using LLMs to their full potential while maintaining effective token usage.<br>You'll find plenty of discussion about prompt engineering and model selection, but the orchestration layer around the model doesn't get nearly as much attention(until recently with a few open source projects being released as discussed later in the post). In my experience, that's where the biggest improvements in capability, cost and reliability come from. A highly capable model with no structure around it will still burn tokens like they're going out of fashion on redundant context and repeat work it's already done, producing results you can't verify or reproduce. Picking the right model and ignoring the harness is like buying a race engine and fitting it to a shopping trolley.<br>What is a Harness?<br>Not that kind of harnessBefore diving into the specifics, it is worth explaining what a harness actually is in the context of AI and LLMs and why it has become one of the most important components of my AI-assisted workflows.<br>A harness is the orchestration layer around an LLM. It controls the inputs, tools, prompts, models, state, validation gates and outputs for each stage of work.<br>If you've read the previous post, MCPs sit inside that layer as one type of tool. They give the model callable functions (run a command, decompile a binary, query a database), but they don't decide when those functions get called, in what order, with what context, or what to do with the result. That's the harness's job. You can have a full suite of MCP servers configured and still produce inconsistent, unverifiable output if nothing is coordinating how they are used.<br>Harness StructureFor offensive security research, that orchestration layer is responsible for making decisions such as:<br>What data should be collected?<br>Which tools should execute and why?<br>Which model is best suited for this particular task?<br>How much context is actually required?<br>How can previous knowledge be reused instead of rediscovered?<br>And perhaps most importantly, when should the model stop thinking and hand control back to the operator?<br>More capable harnesses dynamically select between models, invoke external APIs, execute MCP tools and route between specialised agents. Breaking work into stages like this means the model doesn't need to solve the entire problem in a single pass. The harness provides only the context each stage actually needs, which means lower token burn (or at least that's the plan), more consistent reasoning and less duplicated effort across runs.<br>Side note if you're curious about your token burn and you use Claude, I built TokenBurn for exactly that to map a Claude Max sub to relevant API spend:<br>GitHub - ZephrFish/TokenBurn<br>Contribute to ZephrFish/TokenBurn development by creating an account on GitHub.<br>GitHubZephrFish
Trying to coerce useful work out of LLMs without the harness and verification layer is like herding cats, now if you scale that up to multiple parallel agents and it's closer to supervising a room full of drunk toddlers, each convinced they're helping, none of them checking with each other and falling over the next.<br>I've got eight running in the setup from the previous post(MCPs not drunk toddlers), and the harness is what makes them function as a pipeline rather than a collection of disconnected tools.<br>Before We Begin<br>Now that you're a little bit more up to speed with what a harness is, next up is to explore some of the publicly available ones out there. If you've never looked at harnesses before here are a few worth checking out, all of the ones documented below except Google's Nap Time are open source and mostly actively contributed to allowing you to scan codebases and hunt on security bugs. RAPTOR is the one I find myself using the most but the others certainly have their place if not as a baseline to start from.<br>Harnesses in Practice: RAPTOR<br>GitHub - gadievron/raptor: Raptor turns Claude Code into a general-purpose AI offensive/defensive security agent. By using Claude.md and creating rules, sub-agents, and skills, and orchestrating security tool usage, we configure the agent for adversarial thinking, and perform research or attack/defense operations.<br>Raptor turns Claude Code into a general-purpose AI offensive/defensive security agent. By using Claude.md and creating rules, sub-agents, and skills, and orchestrating security tool usage, we confi…<br>GitHubgadievron
While the idea of an LLM harness might sound abstract, frameworks such as RAPTOR provide a useful example of what this looks like in practice. Rather than relying on a single prompt and expecting a model to discover...