The Review That Praised the Bug | MrJStickel | MrJStickel<br>Back to Projects<br>The Review That Praised the Bug<br>Three frontier models reviewed the same code - then the reviews got graded

Measured 2026-07-30

LLM code reviewClaim-by-claim verificationBM25 / IDFA/B measure-gatingReview forensics<br>The same bundle of my platform's code went to three frontier models from three different companies for review, and the reviews got the same treatment my system's answers get - every claim checked against the code before any of it was believed. One review went five for five and found a real algorithmic bug. One had the right target for the wrong reason, twice. The near-perfect one fabricated features, recited my own portfolio back to me as verification, and rated the buggy function best-in-class. Then the one verified-true finding was implemented, measured twice, and rejected by the A/B - so the strongest finding any of them made did not ship. Reviewed and measured 2026-07-30; the A/B ran on Kin, my private instance.

Reviews of One Bundle<br>three frontier models, three different companies, same code

5/5<br>Best Review, Verified<br>all five findings true - including the only real bug found

Fabricated<br>The Warmest Review<br>caches, retries, and failover paths praised that are not in the code

429<br>The "Missing" Rate Limit<br>reported absent; answers with a 429 in production

76 vs 75<br>The A/B Verdict (of 83)<br>the one true finding lost the measurement and did not ship

Rungs on the Ladder<br>fabricated praise, wrong-reason pointer, verified finding, measured win

The Three Reviews, Graded<br>Same code, same ask. The grades below are against the source, not against each other.<br>The accurate one: five findings, five verified, zero false claims<br>Every finding held up against the source, including the only real algorithmic bug any reviewer caught: the keyword half of hybrid retrieval scores terms by how rare they are, and the code computed rarity over the few dozen candidates already fetched instead of over the whole corpus. At a pool of 60, a term appearing in all 60 candidates scores near zero while a term appearing in one scores 3.7 - the exact inversion of intent, penalizing topic-defining terms precisely when they discriminate most. This review also did what accurate reviews do: it asserted nothing the bundle could not support.

The right target, the wrong reason - twice<br>The second review flagged the exact function that holds the bug, but for a per-query cost problem it does not have - the loop runs over tens of candidates, not the corpus. It also flagged a real weakness through an attack path this deployment does not expose. To its credit, it plainly hedged what the bundle did not show instead of guessing. Its praise was audited too: of four asserted strengths, three held and one - "zero data egress" - was a misleading frame. Embedding and reranking are local, but answers ship retrieved text to a cloud model unless routed to the local lane. The honest claim is scoped, not absolute.

The glowing one: near-perfect score, least reliable review<br>It declared the system better than almost anything in production - and, checked claim by claim, fabricated the most: a cache that is not in the code, retry logic that is not in the code, a database failover path that is not in the code, a description of the vector store that contradicted its own earlier correct statement, and CI scanners plus nightly backups asserted sight-unseen. It reported a missing rate limit that answers with a 429 in production, recited numbers from my public portfolio back to me as things it had "verified" in the source, and rated the one function that actually contains the bug as best-in-class. One recommendation was actively harmful: moving secrets from host-external injection into a file inside the repo tree, dressed as a best practice.

The Coda: A True Finding That Did Not Ship<br>The part most reviews never get - the measurement.<br>The one true finding was implemented and measured<br>The pool-local rarity bug is real - so the accurate review earned a measurement, not a merge. The fix (corpus-wide statistics in the fusion scorer) was built exactly as scoped and A/B tested against the unchanged system on the live corpus: the baseline found the right source for 76 of 83 questions; both versions of the fix found 75. Zero misses converted, one boundary hit lost, rank-1 performance flat.

Why a true finding still lost<br>Architecture ate the bug: the fusion score only orders the wide candidate pool, and a cross-encoder reranker owns the final top-5 - so the pool-local inversion has almost no blast radius on this pipeline. The code was reverted, and the re-open condition is pinned in a regression test: if the reranker ever stops owning the final stage, the finding gets re-measured, not re-debated.

The ladder the week built<br>Fabricated praise, then a wrong-reason pointer, then a verified-true finding, then a measured improvement - and none of the three reviewers reached the top rung. Review verdicts...