The Prover, The Skeptic, and The Judge – ‎[INSERT CHAOS]

Point an autonomous LLM-powered security agent at a target and tell it to find vulnerabilities, and it will find vulnerabilities. It finds them...">

Point an autonomous LLM-powered security agent at a target and tell it to find vulnerabilities, and it will find vulnerabilities. It finds them...">

Point an autonomous LLM-powered security agent at a target and tell it to find vulnerabilities, and it will find vulnerabilities. It finds them...">

The Prover, The Skeptic, and The Judge

11 Jul, 2026

Point an autonomous LLM-powered security agent at a target and tell it to find vulnerabilities, and it will find vulnerabilities. It finds them every time. Point it at a hardened, freshly patched box with nothing actually wrong on it, and you still get a report back, formatted nicely, seemingly suitably alarmed, impact ratings and everything. Unfortunately, current models are biased towards generating plausible analyses rather than confidently concluding that no vulnerabilities were found, particularly when prompts and agent workflows implicitly assume there is something to discover. The result can be nice-looking reports containing convincing but nonexistent vulnerabilities.

So you end up with a pile of findings and no quick way to know which ones are real because the actual bugs and the completely invented ones show up looking identical.

When confidence tells you nothing

An example of a finding an agent produced not long ago:<br>SQL Injection In /login Endpoint

Impact: Critical

It even provided a PoC python script that, according to the transcript, ran and came back with a Success message.

Maybe it's genuine? Or maybe the endpoint returned a 500 for some unrelated reason and the model read "error in the response" as "injection confirmed". Maybe the script printed Success but nothing in it ever actually checked whether it actually succeeded. Maybe the whole thing's a convincing hallucination, stitched together from the shape of a few thousand security writeups the model saw during training. You can't tell which of these happened by looking at the report, because the report reads exactly the same either way.

It gets worse when you look at where the finding came from. The first pass is usually run on a fast, cheap model, because maybe you want to cover as much surface as possible without spending a fortune, but the fast cheap models are the ones that over-report and over-dramatize, and from what I have seen, have little idea how to correctly rate a finding with respect to the overall context or business purpose of the application. They pattern-match to the shape of a vulnerability and file it. So the stage that produces the most findings is also the stage you should trust the least, which is a bad combination when the report from that stage is the thing that could land on someone's desk.

The obvious response is to write a firmer prompt:<br>Only report vulnerabilities you've verified.

Don't report false positives

(which never sticks anyway).

You're asking one agent, working in one train of thought, to be the eager hunter who turns up bugs and at the same time, be the hard-nosed critic who throws most of them out. Those two want different things but they're living in the same head, and the hunter tends to win because turning something up feels like getting somewhere, and admitting you were wrong feels like walking backwards.

One solution is to stop asking a single agent to do both.

So we split the work across roles that don't share a train of thought and, more to the point, don't share a motive.

The Prover

The Prover does the thing the agent was always going to do. It pokes at the target, comes back with a bug candidate, and writes it up as a claim with evidence behind it. SQL injection at /login, critical impact, here's the poc.py that shows it. The Prover gets to be eager. That's what it's wired for. Nobody's trying to talk it into caution. The goal is to let it be productive and then send whatever it produces somewhere that isn't.

The Skeptic

That somewhere is the Skeptic. The Prover can run cheap, because turning up candidates is the easy part. The Skeptic runs on a stronger model with a bigger reasoning budget. It takes the Prover's claim and the evidence and goes after them hard in an attempt to knock the thing down.

The Judge

The Judge comes last. It doesn't run anything itself. It reads the Prover's case and the Skeptic's objections side by side, weighs them against a fixed set of questions about evidence, reproducibility, impact, and confidence, and rules. It's the only role whose whole job is to say no and mean it, and because it never had the fun of finding the bug, it has nothing invested in the bug being real.

Written out, the instruction that drives this is blunt about the three rounds and about not letting the first one stand:

MODE: panel. Treat each finding as an adversarial match played in three<br>rounds, and play all three in...