The Harness Is Not the Model: How Far Scaffolding Takes a Weak LLM

Skip to content

Projects<br>The Harness Is Not the Model: How Far Scaffolding Takes a Weak LLM

July 5, 2026<br>Abstract<br>I made an engineering bet: wrap a swappable weak model (local on a Laptop, or a cheaper API tier) in the strongest possible harness, deterministic retrieval from a personal knowledge base, a curated toolset, verification loops, and expect frontier-level behavior from the assembly. This paper formalizes that bet into five testable components and tests each against roughly 30 published sources from 2022 through mid-2026. The evidence splits cleanly. The harness closes the gap almost fully on knowledge (an 11B retrieval-augmented model beating a 540B model on NaturalQuestions), on single-turn tool-call syntax, and on narrow, cheaply verifiable tasks (a roughly 15x swing on SWE-bench with weights held constant), buying roughly one model generation, a 2 to 6x multiplier from a naive baseline. It then hits a model-bound ceiling: same-harness model swaps produce 1.5 to 4x spreads on agentic coding and 50 to 100x spreads on METR autonomous time horizons, multi-turn tool coherence collapses with model size, and self-correction without an oracle degrades output. I concede the components that fail, keep the ones that survive, and end with the architecture the evidence supports: tiered routing with frontier escalation, and a rule of thumb that once a competent harness exists, swapping the model dominates every other intervention.<br>Keywords: harness engineering, LLM scaffolding, test-time compute, agentic coding, function calling, local inference, tiered routing, model cascades.<br>1. Introduction<br>1.1 The Bet<br>The plan was concrete. I maintain a personal knowledge base, a vectorless Markdown wiki with deterministic two-step retrieval, and I wanted to connect it to a swappable local model running on a Laptop. The brain supplies the facts. A curated toolset supplies the hands. A harness (structured prompts, staged workflows, verification gates, retry logic) supplies the discipline. The model in the middle becomes a commodity: swap in whatever runs on the hardware this month, and the assembly around it produces frontier-level thinking and execution anyway. If the bet held, I would own the whole stack, pay no per-token rent for routine work, and upgrade by config change.<br>Call this position harness maximalism. I believed a strong version of it, and my own systems gave me reasons to. I had watched a hierarchical multi-agent setup cut per-module delivery time by roughly 55 percent across more than 48 measured sessions, and I had watched a Haiku-grade prompt-writer model, given a deliberately narrowed task, cut sub-agent token consumption by 40 to 60 percent and drop retries from 1.5 to 0.6 per task. Scaffolding visibly multiplied output. It was tempting to extrapolate: if structure buys this much at the top tier, surely enough structure lifts the bottom tier to the top.<br>This paper is the test of that extrapolation. I went looking for published evidence that would either confirm the bet or break it, and I report both outcomes honestly.<br>1.2 Formalizing the Bet<br>"Frontier-level performance" is not one claim. It bundles capability dimensions that behave differently under scaffolding, so the bet decomposes into five testable components:<br>Knowledge : retrieval substitutes for what the model knows.<br>Format reliability : schemas and tuning substitute for native instruction-following on tool syntax.<br>Narrow verifiable tasks : scaffolding plus verification substitutes for capability wherever an automatic check exists.<br>Judgment and coherence : harness structure substitutes for the model's ability to hold state and reason across many turns.<br>Long-horizon autonomy : orchestration substitutes for the model's ability to sustain a multi-hour task.<br>Table 1 states the verdicts up front; the evidence draws the line between component 3 and component 4.<br>Table 1. The closability matrix: each component of the bet, the harness element that bears on it most, and the verdict the published evidence supports.<br>#Component of the betPrimary harness elementVerdictAnchor evidence1KnowledgeRetrievalCloses, almost fullyAtlas 11B beats PaLM 540B (Izacard et al. 2022)2Format reliabilitySchemas plus tuningCloses, single-turn onlyToolACE-8B above GPT-4-class on BFCL single-turn (BFCL 2026)3Narrow verifiable tasksVerification loopsMultiplies enormously, given an oracleRoughly 2 percent to 33.2 percent, same weights (Yang et al. 2024; OpenAI 2024)4Judgment and multi-turn coherenceOrchestrationModel-bound; decomposition mitigates, does not closeBFCL multi-turn collapse by model size (TinyLLM Survey 2025)5Long-horizon autonomyOrchestrationModel-bound; the widest gap of all50 to 100x METR time-horizon spread (METR 2025; METR 2026)allCostRoutingCloses almost completely98 percent cost reduction at matched accuracy (Chen et al. 2023)<br>Sections 2 and 3 walk the cells. Section 4 grounds them in what...