The AI PR Security Review Frontier | Dam Secure Docs

Skip to main content<br>We wanted to take the guesswork out of which model to use for PR security reviews. To our surprise no Anthropic model reaches the frontier at all and the latest GPT5.6, released only 3 days ago, is the king. Although Fable is known to perform well on a full-code scan, PR reviews are a common workload for our users.

This chart plots Cost per Pull Request vs F1 (finding quality). Blue points define the frontier. See metrics definitions in the table below.

Summary​

We ran 10 models over the same 10 pull requests, each carrying one planted access-control bug (IDOR, missing auth, broken authorization), five times per model, and scored every finding against the actual code.

The result is a clear cost/quality frontier, and it belongs to OpenAI, xAI and Google rather than Anthropic. GPT-5.6 Sol dominates. Fable 5 performs poorly at ~$3.61 per PR, lands lower on both quality and value. No Anthropic model reaches the frontier at all.

Our benchmark repos are synthetic and private to avoid Benchmaxxing and accidental inclusion in model training.

NOTE:

Fable does perform well on Full Code Scans, but this workload is a PR Scan. A PR Scan is a very common workload for our Users.

This is not a quirk of the Dam Secure harness. We tested the Pydantic harness and Claude Code harness as well and got very similar results.

Benchmark Scope: Pull Requests Only​

This benchmark is only trying to measure how good models are at finding vulnerabilties in Pull Requests where additional code has been added. This benchmark is NOT about extensively searching an existing codebase for all known vulnerabilities. Our full scan benchmark is coming soon.

That scope matters for how you read the results. The key takeaway is that you are wasting money, usage limits, and tokens if you reach for Fable (or other frontier Claude models) to find security issues in PRs. Fable 5 is the most expensive configuration we tested and still lands off the cost/quality frontier — dominated on both axes by GPT-5.6 Sol, which reaches higher recall at roughly a fifth of the cost per PR.

We will separately release another article on how these agents perform when freely roaming an existing large codebase. The model rankings may look different there; this post is about PR review only.

Results​

Each run used the Dam Secure Vulnerability Scanner with high reasoning, repeated five times on a 10-PR corpus of planted access-control bugs.

ModelRecallPrecisionF1Cost / PRCost / TPGPT-5.6 Sol via OpenRouter100%83.3%0.91$0.70$0.70GPT-5.5 via OpenRouter94%83.9%0.89$1.24$1.32Fable 5 → Opus 4.8 fallback88%83%0.85~$3.61~$4.10Claude Sonnet 4.680%85.1%0.82~$1.22~$1.53Gemini 3.5 Flash84%76.4%0.80~$0.94~$1.12Grok 4.5 via OpenRouter74%80.4%0.77$0.20$0.27Gemini 3.1 Flash Lite68%82.9%0.75~$0.04~$0.06Claude Opus 4.860%81.1%0.69~$1.72~$2.87Claude Haiku 4.556%73.7%0.64~$0.75~$1.34DeepSeek V4 Pro via OpenRouter44%81%0.57$0.14$0.31

True Positive (TP) - a vuln found that was expected to be found.

False Positive (FP) - a vuln found that was not expected (not a real bug).

False Negative (FN) - a vuln not found that was expected to be found.

Recall (R) - how well did it find the planted vulnerabilities? It is the fraction of actual positives correctly identified: TP / (TP + FN).

Precision (P) - false positive performance. A fraction of predicted positives that are correct: TP / (TP + FP).

F1 - the overall performance balancing Recall and Precision: 2PR / (P + R). Hovering a point also shows F2 = 5PR / (4P + R), which weights recall ~4× precision.

Costs / PR - is the cost to run that model over all prompts and process the PR regardless of it's accuracy. This is the most accurate representation of real-world cost.

Cost / TP - is the cost per true positive found.

Wikipedia: Precision, Recall and F1

Results Commentary​

GPT-5.6 Sol is the first model to reach 100% Recall on our harness (clearly we now need to add harder vulnerabilities to find).

The Fable 5 row in the table is labeled "Fable 5 → Opus 4.8 fallback". It handed off to Opus 4.8 only 10.7% of the time.

Prior to GPT5.5 coming out, Gemini 3.5 Flash has been our preferred model for this type of workload. This has previously been our little secret, because our evaluation suites have proven despite the industry wanting to believe that the Opus models dominate security.

Best model is GPT-5.6 Sol - strong F1 (0.91), perfect 100% recall and very cost effective at $0.70 per PR. 5.6 is ~45% cheaper than it's predecessor 5.5 for roughly the same performance.

Honorable mentions: Grok 4.5 lands on the frontier at $0.20 per PR with F1 0.77 (74% recall, 80.4% precision), and Gemini 3.1 Flash Lite anchors the cheap end at ~$0.04 per PR with strong results: F1 0.75 (68% recall, 82.9% precision).

We'll add other Open-weight models to this benchmark.

Methodology​

We construct synthetic pull requests from open source repositories. Each pull request has a...