We Debug Our AI Harness on Weak Models on Purpose | Blog | Archestra<br>Star us on GitHub ⭐<br>Star us on GitHub ⭐

Written by<br>Arseny Kravchenko

Strong models are excellent at hiding product bugs. Give one a broken tool call or a confusing error, and it may still finish the task.

Weaker models are less forgiving. When the tools, prompts, or runtime are wrong, they fail quickly, which makes them useful for debugging. I think of it as the difference between testing an app on a maxed-out MacBook Pro and testing it on an old ThinkPad.

We run Archestra Chat (our built-in chat interface for working with agents and MCP tools) on weaker models on purpose, record the full trajectory, and investigate the failures they expose. This approach has found defects in file handling, sandbox tooling, provider schemas, and our agent runtime. Fixing those bugs helps every model - not just the weaker ones. Strong models no longer need to spend extra tokens and retries working around our mistakes. As a result our harness works reliably even with weaker models (not only models capable of compensating for its flaws).

The idea came from a customer who asked me a simple question: would Archestra Chat work with cheaper models? I couldn't answer. At the time, we were testing changes one at a time, adjusting tools and prompts, and checking whether the final answer looked right. Sometimes it worked. Sometimes it didn't. We had no repeatable way to measure it. I'm an ML person at heart, and when a system behaves like that, the first thing I build is a validation loop.

That loop became an end-to-end benchmark based on 26 real customer workflows. It starts the actual product, runs each task, and checks the answer against hidden ground truth. Every night, it repeats the same tasks across about ten models. Important to note: this is not an academic benchmark attempt. But it's an advanced integration test that answers two questions: where does our product fail, and how far down the price ladder can we go before the assistant stops doing useful work?

How we benchmark AI agents on real work, not party tricks

I approached this the way you'd approach any serious evaluation: the test has to be something the system can't cheat, and it has to look like reality. That leads to two design pillars.

First, the assistant submits its final answer through a dedicated channel. This channel checks the format, not whether the answer is correct. If the task expects a number, it rejects text. If it expects three candidates, it rejects two.

The answer is checked for correctness only after submission, using deterministic code, not another LLM. The grader and expected answer are hidden from the agent during the run. If the agent could read the answer key, it could game the test, and then the score would be a lie. It can't, so the score means something.

The boundary that makes the score mean something:

Loading diagram...

submit_result never tells the agent whether it was right - only whether the JSON parsed against the task's schema. The expected ground truth lives entirely on the right of that line and is never staged into the sandbox, so the agent has nothing to read off.

Second, we don't record just pass or fail. We capture the assistant's entire visible trajectory: every message, tool call, file it touched, and artifact it produced. That turns “it failed” into “here's exactly where it failed.” It's the difference between a number that shames you and a report you can actually use. More on that shortly, because this became the most useful thing we built.

The tasks come from reality, not our imagination. Each one is based on something we actually saw: a real customer workflow or a real incident, lightly anonymized. We have a small set today, and it grows whenever reality hands us a new headache. Once a task goes in, it stays in, a permanent tripwire for that failure mode.

A few examples, to make this concrete:

Document workflows. Approve invoices and shortlist CVs from mixed file formats while rejecting hidden prompt injections and inconsistent data.

Incident triage. Find the cause of an outage in a zip of unsorted logs containing plausible red herrings.

Memory across conversations. Create a file in one conversation, then retrieve and use it in a new one - this exercises the persistent files we shipped in July.

Live facts. Fetch a repository's current star count, an asset price at a specified time, or the latest package version.

We've tightened them to match how a real employee actually talks: “Here are some invoices. Do the right thing.” That harder version is the one worth measuring.

We also keep a hard boundary around what belongs in the suite. Tasks should be difficult, but solvable with the tools Archestra Chat actually has: file access, web access where needed, the sandbox and skills, and the customer's connected systems. We're not measuring whether a model can do frontier physics, and we're not testing features we haven't built yet. The useful...