We scanned 1,868 AI-built apps, and audited our own scanner | PathToShip

This is the post behind every number we publish about AI-built code. It has<br>two halves, and the second one is the uncomfortable one. First: what we found<br>scanning 1,868 public apps built with AI coding tools. Second: how we<br>audited our own scanner to make sure those numbers are honest, including the<br>part where we discovered we'd been over-counting.

The short version: most scanners never publish their false-positive rate.<br>We measured ours on a representative sample of our critical findings, and it<br>was about 42%. We fixed the classes behind it, re-scanned all 1,868 apps, cut<br>the rate to about 25%, and the corrected numbers are below.

If you arrived from a headline, this is where it has to defend itself.<br>Definitions are pinned, caveats come before results, and the places our<br>scanner gets things wrong are documented, not buried.

What we scanned

We targeted roughly 1,900 public GitHub repositories built with AI coding<br>tools. A handful were unreachable (deleted, emptied, or timed out), leaving<br>1,868 analyzed .

For tool attribution we use hand-checked labels first (repos we identified as<br>built with a specific tool from clear evidence, like the tool's own config<br>files or generated artifacts), and fall back to the scanner's automatic<br>detection only where no label exists. Every per-tool number below uses that<br>one method.

Nine tools, each with a healthy sample: Bolt, Lovable, v0, Cursor, Replit,<br>Tempo, Base44, Firebase Studio, and Windsurf (per-tool counts in the table<br>below), plus 234 repos no method could attribute.

One transparency number before any results: our automatic detector now agrees<br>with the hand-checked labels 92% of the time, up from 76% at our first<br>report, after we fixed a detection bug (more on that below). Detecting which<br>tool built an app is genuinely hard, because some editors leave almost no<br>trace, which is why the table leans on hand-checked labels rather than the<br>detector alone.

What the scanner checks

PathToShip runs 80 automated checks across 7 weighted dimensions: Security<br>30%, Production readiness 18%, Architecture 12%, Scalability 12%,<br>Infrastructure 12%, Code quality 8%, Cost efficiency 8%.

Severity sets the penalty: critical −30, high −14, medium −6, low −3, and<br>info-level findings always carry zero penalty: they're observations, not<br>score movers. Two design choices worth knowing:

Empty repos don't win. A dimension with nothing to evaluate scores 50,<br>not 100. The scanner does not reward having nothing to find.

The production-ready bar is 80/100 , the same threshold our certification<br>uses: one bar, applied identically to every repo,<br>including our own.

The honest scope statement: this is pattern-based static analysis over a<br>capped download of each repo (200 files / 500KB, which keeps a scan around 30<br>seconds). It cannot see runtime behavior, billing plans, deployment settings,<br>or how data flows between files. Where that bit us, we say so.

Selection bias, stated plainly

These are public repos. People who build serious production apps tend to<br>keep them private. Our set almost certainly over-represents hobby projects,<br>demos, and learning exercises, and under-represents AI-assisted apps with real<br>users and real revenue.

So every statistic here is "of the repos we scanned." We make no claims about<br>AI-built apps in general, and if you quote one of our numbers that way, you're<br>quoting it wrong.

The results

Score distribution: mean 68.3, median 71, range 31–95. Most of the set<br>lands in the working-but-gapped middle band: shippable demos, not shippable<br>products.

Headline rates, of the 1,868 repos scanned, definitions pinned:

23% pass the 80-point production-ready bar.

24% have at least one critical finding , after the accuracy work<br>described in the next section. (The pre-audit scanner said 33% on this same<br>set; hold that thought.)

15% ship a hardcoded secret or API key , defined as the union of our<br>"Hardcoded Secret/Password" and "Hardcoded API Key" rules, counted per repo.

35% have an XSS-class finding , defined conservatively as the union of our<br>two dangerouslySetInnerHTML rules (a broader definition that also counts raw<br>innerHTML gives 43%).

By dimension , the pattern is the whole thesis: production readiness is the<br>weakest (mean 56), then security (65) and architecture (63); scalability (80)<br>and cost efficiency (88) are strong. AI tools generate efficient code that<br>isn't ready to operate. What's missing is the secrets handling, the error<br>fallbacks, the logging, and the deployment hygiene.

The per-tool table

The framing that has to come before the table: this is a snapshot of public<br>repos per tool, not a tool ranking. Different tools attract different users<br>building different things. Treat the rows as observations, nothing more, and<br>read the next section before you take any single number to the bank.

ToolReposMedian scoreMean security% with ≥1...