What a Verification Loop Adds to a Coding Agent: A First Look | by IronBee | Jul, 2026 | MediumSitemapOpen in appSign up<br>Sign in

Medium Logo

Get app<br>Write

Search

Sign up<br>Sign in

Introduction<br>Methodology<br>ResultsAnalysis 1: How far each arm gets<br>Analysis 2: Same result, different price

Closing<br>References

What a Verification Loop Adds to a Coding Agent: A First Look

IronBee

12 min read·<br>Just now

Listen

Share

This is the opening post in an ongoing series. We start with one model pair on one project, and the analysis will continue across more models and more datasets. We are sharing early on purpose, and we will keep sharing as we go.<br>Press enter or click to view image in full size

Introduction<br>A coding agent can produce a lot of code quickly. What it cannot do, on its own, is know whether that code works. The model writes something plausible, the run moves on, and any mistake travels with it. On real multi-step projects this compounds: one unverified error early on can sink everything built on top of it. The fix human teams use is old and boring. Someone checks the work, against the running application, before anyone builds on it.<br>This post asks one question: how much does adding that check change what a coding agent can finish? Our hypothesis going in was that a closed verification loop (run the app, exercise the change, find the failure, drive the fix, then prove the repair) matters more than raw model strength for a large class of failures. Code is not done until it is verified.<br>To test this we needed a concrete verification layer, and we used our own. IronBee is a verification and fix layer for AI coding agents: when an agent writes code, IronBee opens the app in a browser, exercises the change, and looks for problems. When it finds one, it analyzes the failure, finds the root cause, drives the fix, and then re-verifies, so a fix counts only once it is proven. We built IronBee, so read this as a disclosed-interest experiment with an open method. In this post IronBee is the instrument, not the subject: the conclusions we care about are about verification loops in general, and the setup is public so anyone can rerun or challenge it.<br>We spent a week building a careful test, and we are sharing the opening results now, with more to come. We took real web development work and ran two models, one low-cost and open, one a commercial frontier model. Setting this up well was harder than it sounds, and the next section explains why. We treat this as a beginning, not a verdict. The point of the series is to see how the picture changes as we add more models and more kinds of work.<br>Methodology<br>Choosing the right dataset. We wanted work that looks real, not toy problems. We picked Web-Bench, an open benchmark from ByteDance. It is public, so anyone can check our setup: the code is on GitHub, the dataset is on Hugging Face, and the design is described in the Web-Bench paper. Web-Bench has 50 projects. Each project is a single web app assembled over 20 tasks that must be done in order, so each task builds on the one before it. That sequential structure is exactly why we chose it over a benchmark of independent, one-off tasks: a verification loop earns its keep when work builds on earlier work, so an unverified error compounds instead of staying local. Every task has a hidden test that checks the result, and the agent never sees it. If a task fails, the run stops there. This keeps the benchmark honest: an agent cannot skip ahead or hide a broken step.<br>For this first work we use one of these projects, the survey app, which builds a form with questions, required fields, and a preview. It spans the difficulty range we cared about, from a simple form to logic that has to survive the round trip between the design page and the preview. It is also a good match for what we set out to measure. A verification loop pays off while the code is being written, on exactly this kind of failure: code that reads as correct on its own but only breaks when the running app is exercised. So the lift we see here reflects what verification actually adds during real development, not an artifact of an easy or contrived task. Adding more projects is one of the first things on the list.<br>Choosing the models. We started with two. DeepSeek (deepseek-v4-pro) is a strong model with a low price, so it stands in for the low-cost coder we want to lift. Opus (claude-opus-4–8) is a frontier model, and it sets the quality bar. Putting them side by side lets us ask a clear question: can a verification loop bring a low-cost model up to the level of a frontier model? We built the harness so we can run several models through the same path, and this pair is where we begin. One note on the setup. In this first run DeepSeek is text-only, so it cannot see the screen. That is fine here, because the checks read the page as structured text, through the accessibility tree, the DOM, and the console, not as an image. The model reaches its result on structured...