Benchmarking 15 LLMs on SAST false-positive triage

New: Pressure test your SOC 2 report against buyer requirements<br>Free check

Solutions

PricingResources

About

Log In<br>Book a Demo

We benchmarked 15 models on real security findings. Kimi K3 won, with one caveat that matters

We benchmarked 15 LLMs on 142 real security findings to triage SAST false positives. Kimi K3 led on accuracy, with one caveat that matters for security.

SAST tools would rather flag ten harmless lines than miss one real vulnerability. We built an LLM triage layer that knows how to say "I'm not sure."<br>By Anže Pečar, Founding Engineer & Tim Olshansky, Co-Founder, Fencer<br>SAST tools are tuned for high recall (flag ten harmless lines rather than miss the one real vulnerability). The result is predictable: teams learn to ignore the scanner, and the one finding that matters gets buried with everything else.<br>At Fencer we run an LLM triage layer over our scanners' output: each finding gets read in context and classified as a real vulnerability or a false positive. We benchmarked fifteen models: Claude Opus 5, Opus 4.8, Sonnet 5, and Haiku 4.5; GPT-5.6 Sol, Terra, and Luna; Kimi K3; and a stack of open-weight models, all against 142 findings our engineers had already reviewed.<br>Kimi K3 came first on accuracy , clearing more false positives than any Claude or GPT model we tested. It's also the only model near the top of the table that dismissed a real vulnerability as a false positive . In security triage, that's the one mistake you can't shrug off, and it's why we're not switching to it today.<br>How we built the test<br>Hand-labeling a few dozen findings is quick but noisy. Instead we derived ground truth from what actually happened in production : engineers' thumbs-up/down on past triage verdicts, findings dismissed as false positives, and findings that got fixed. 142 findings total: 11 confirmed vulnerabilities and 131 false positives.<br>That ratio reflects reality. Real SAST output is overwhelmingly noise, so a triage model has two jobs: clear the false positives (131 chances to be useful) and never quietly drop a real vulnerability (11 chances to do real harm).<br>Following the LLM4SA approach, each finding goes to the model with its sink, callers, callees, and taint sources, the same evidence a human reviewer would read before making a call. Each finding is classified three times; majority vote wins. The model is instructed to be conservative : when it can't rule the vulnerability out, it returns uncertain and the finding goes to a human.<br>We score three outcomes on the confirmed vulnerabilities: caught , uncertain (escalated to a human), and dismissed (silently closed as a false positive). The distinction between uncertain and dismissed is the whole point of the design.<br>The results<br>ModelAccuracyVulnerabilities caughtVulnerabilities dismissedFalse positives clearedUncertainCost / findingKimi K3 92% 9 / 111 122 / 131 5$0.052Claude Opus 4.892%10 / 110120 / 1311$0.092Claude Sonnet 589%11 / 11 0116 / 1314$0.047Claude Opus 588%*8 / 9*0117 / 122*3$0.074GPT-5.6 Luna82%11 / 11 0106 / 1312$0.010Claude Haiku 4.581%8 / 111 107 / 13113$0.012gpt-oss-120b80%9 / 110105 / 13120$0.002Mistral Devstral 278%8 / 110103 / 13123$0.003MiniMax M2.576%11 / 11 097 / 13125$0.005GPT-5.6 Terra75%11 / 11 096 / 13111$0.024GPT-5.6 Sol73%11 / 11 093 / 1319$0.051Kimi K2.573%5 / 111 99 / 13132$0.006Z.AI GLM-572%10 / 11092 / 13131$0.008NVIDIA Nemotron Super 371%11 / 11 090 / 13129$0.003DeepSeek v3.263%9 / 11081 / 13130$0.007*Opus 5 scored on the 131 of 142 findings it would answer (see below).<br>Kimi K3: a real jump, and a real flaw<br>The headline isn't K3's top spot, it's the gap to its predecessor. Kimi K2.5 finished twelfth at 73% , catching 5 of 11 real vulnerabilities and returning uncertain on 32 findings. K3 sits at 92% with 122 of 131 false positives cleared and only 5 uncertain. Largest generational jump we measured from any vendor.<br>It wins on the axis that matters most for noise: 122 of 131, two ahead of Opus 4.8 and six ahead of Sonnet 5. And it's decisive: only 5 uncertain, where most mid-tier models punt 20 to 33 findings back to a human.<br>Then there's the caveat. On one finding, an HTML fragment loader that writes a fetched response straight into innerHTML, K3 was the only model in the entire lineup to rule it safe. All fourteen others flagged it or escalated it. K3 dismissed it unanimously across all three runs, and its reasoning shows exactly how:<br>"the response body is trusted same-origin server output, not direct attacker input; any user data within it must be escaped by the server-side renderer "<br>K3 didn't miss the sink. It spotted it, then talked itself into assuming a protection it couldn't verify. "The server must be escaping this" is an assumption, not evidence.<br>Opus 4.8 and Sonnet 5 dismissed zero real vulnerabilities . Sonnet 5 caught all 11. When those models can't confirm that data is safe, they flag or escalate. K3 sometimes decides on its own.<br>On a queue that's...