We Scanned 27,075 Real Developer Prompts to ChatGPT and Found Three Live API Keys | HeimWall<br>For engineersFor teams

Somewhere in 2023, a developer did what the median developer now does several times a day: they pasted a Python file into ChatGPT. The file contained a live OpenAI API key — the full 51-character sk-... string, sitting right there in the source.

The conversation got shared. The share link got committed to GitHub. Researchers scraped it into a public dataset. And this month, our detection engine flagged it — three years after the paste.

That key is one of three genuine, authentic-format API keys we found in a single weekly snapshot of real ChatGPT traffic. Not synthetic test data. Not a red-team exercise. Real developers, doing real work, leaking real credentials — exactly the behavior HeimWall exists to catch at the source.

This post is the full readout: the dataset, the method, the numbers, and — because we hold ourselves to naming limitations before anyone else does — the false positives and the things this benchmark cannot tell us.

The dataset: real prompts, not lab prompts

Most secret-detection benchmarks are built from labeled corpora: known secrets planted in known positions, so you can compute precision and recall. Those are useful, and we run them. But they share a weakness — the input is curated. It tells you how a scanner performs on secrets, not how it behaves on a Tuesday.

DevGPT is different. It's the MSR 2024 Mining Challenge dataset: ChatGPT conversations that real developers shared publicly, scraped from GitHub and Hacker News. We used the final weekly snapshot (October 2023) and scanned user prompts only — the direction of traffic HeimWall actually guards. The tally:

3,559 source records, containing 4,733 shared ChatGPT conversations

27,075 developer prompts scanned

Prompts referenced from files (21,089), issues (2,027), commits (1,602), Hacker News threads (1,139), pull requests (996), and discussions (222)

One thing DevGPT does not have: ground-truth labels for sensitive content. Its annotations describe software artifacts, not secrets. So this benchmark cannot produce a recall number — and we won't pretend otherwise. What it measures is something the labeled corpora can't: HeimWall's firing rate on genuine production-shaped input , followed by manual triage of everything that fired. In other words: how noisy is this thing in the real world, and what does it actually catch?

That's the alert-fatigue question, and for an inline security tool it's the question that decides whether the tool survives contact with engineers.

The headline number: 1.12%

Across 27,075 real prompts, HeimWall raised at least one alert on 303 of them — a firing rate of 1.12% , or roughly one alert per 89 prompts.

By severity:

SeverityCountRateCritical110.041%High910.336%Medium700.259%Low1310.484%<br>For context on what that means in a workflow: an engineer sending 30 prompts a day would see an alert roughly once every three days, and a critical alert a few times a year. That is a friction budget an engineer will tolerate. Our own principle — soft notifications beat hard blocks in 19 out of 20 cases — only works if the notification is rare enough to mean something. 1.12% clears that bar.

The firing rate also varies with context in a way that matches intuition about when developers leak:

Artifact contextPromptsFlaggedRatePull requests996191.91%Issues2,027301.48%Files21,0892321.10%Commits1,602150.94%Hacker News1,13970.61%Discussions22200.00%<br>PR and issue contexts fire at 1.5–1.9% — the debugging surface, where developers paste error output, config, and failing records wholesale. The dataset confirms the intuition at scale: the leak risk concentrates where the developer is stuck and pasting fast.

What was actually in there

Firing rate is the operational number. The security question is what's inside the flagged set. Since there are no labels, we manually classified every flagged item, deduplicating by matched value (one leaked key tends to recur across a conversation). Everything below is manual verification, not automated ground truth.

Confirmed leaked secrets: 3 distinct live-format API keys. (The same key can appear in several flagged prompts; these are unique values after deduplication.)

Two OpenAI API keys — authentic 51-character sk-... format, pasted inside Python source

One Google API key — authentic 39-character AIzaSy... format, sitting in an Expo app.json

(Our Slack-token rule also fired four times — all four collapsed to a single xoxb-your-token placeholder. False positive, counted as such.)

Personal data: 49 real personal email addresses. Of 62 distinct flagged emails, 49 were genuine personal or institutional addresses — Gmail accounts, company domains — pasted into prompts alongside code and logs. The other 13 were machine-generated or example addresses.

Three keys and forty-nine emails may sound small. Consider the denominator. This is one weekly snapshot of...