Detecting Claude and ChatGPT using letter counting. | atomic14

🌈 ESP32-S3 Rainbow: ZX Spectrum Emulator Board!<br>Get it on Crowd Supply →

Detecting Claude and ChatGPT using letter counting.

View All Posts

read

Want to keep up to date with the latest posts and videos? Subscribe to the newsletter

Posts ·<br>Videos ·<br>ESP32 ·<br>Tools ·<br>Support

« Detecting Claude by counting letters

HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi

Last time I built a detector that counts<br>four-letter runs (n-grams) and tells you whether Claude Opus 5 wrote something. It works well - about 91% on<br>text it has never read.

It has one obvious hole: it only knows one robot (Claude Opus 5). Paste ChatGPT output<br>into it and it says “meatbag”.

So I’ve fixed that. I’ve added a ChatGPT detector and you can<br>try it here.

Getting the ChatGPT data

We did the same as last time, have the model rewrite the same 519 human passages, three ways. That’s 1,557<br>rewrites, plus 519 passages written from scratch that I keep out of training and use as a test.<br>2,076 generations in total.

Last time this went through Anthropic’s batch API and cost about $30. This time I just ran the command<br>line codex using the subscription I’m already paying. So it didn’t cost anything extra (I’m wishing I’d done<br>that with claude last time!),

There is a small issue with this - coodex is a coding agent, so it will have a system prompt that will<br>influence the output, so our trained up detector might not do as well on text from the ChatGPT app.

The ChatGPT detector

We built this in exactly the same was as the Claude detector - and it works really well!

Detector<br>Cross-validated<br>Held-out authors

Claude Opus 5<br>0.906<br>0.925

ChatGPT<br>0.930<br>0.935

ChatGPT seems to be easier to detect than Claude. Not by a lot, but consistently.

It also seems to handle shorter text better than the Claude detector.

Letters<br>Claude detector<br>ChatGPT detector

100<br>0.645<br>0.709

200<br>0.740<br>0.798

432<br>0.783<br>0.885

1,000<br>0.858<br>0.952

At a thousand letters the ChatGPT detector is on 0.952 against 0.858 for the Claude one, and it’s<br>ahead on short text as well.

Do the two detectors work on each other?

Detector<br>Reading its own model<br>Reading the other one

Claude<br>0.906<br>0.792

ChatGPT<br>0.930<br>0.800

Each one is clearly best on the model it was trained on, but neither is terrible on the other - 0.79<br>and 0.80, against 0.5 for a coin toss.

A third detector

Can we build a detector that does Claude or ChatGPT?

We trained s third detector from scratch on all the data from both models.

Detector<br>ROC AUC<br>Held-out authors

Was this a robot?<br>0.897<br>0.911

Does it look like Claude?<br>0.906<br>0.925

Does it look like ChatGPT?<br>0.930<br>0.935

They’re three separate questions rather than slices of one pie, so they won’t add up to anything. A<br>passage can read as 0.85 robot while both model scores sit lower, which just means it looks machine<br>written without strongly resembling either one.

What it still can’t do

Everything from last time, plus a couple of new ones.

It knows two robots. Gemini, Llama, DeepSeek, anything else - it has never seen them. A low score is<br>not evidence that a person wrote something.

It still detects a style rather than a model. The most useful result from last time hasn’t<br>changed: a writing style held out of training scored 0.153, meaning the detector confidently rated it<br>as more human than the humans. Both new detectors train on all three instruction styles for that<br>reason, but there’s nothing magic about three - we need to add more data.

It’s still for education and amusement, and it’s still not evidence. Please continue not accusing<br>anyone of anything based on my stupid website.

Try it out

meatbag.atomic14.com - paste something in and see which robot it<br>thinks you are.

Related Posts

Detecting Claude by counting letters - Anthropic now watermarks Claude, but I wanted a detector we can run ourselves—so I built one. It ignores words entirely, counts 4‑letter n‑grams, feeds them to a linear SVM, and reliably flags Claude Opus 5 (around 0.90+ ROC AUC, up to 0.977 on fresh writing), even on unseen authors and external benchmarks. It runs in your browser, and along the way it surfaces neat stylistic tells—think Germanic vs. Latinate vocabulary—so you can see why the call was “robot” or “meatbag.”

Don't blame Claude - It's me, I'm the problem, it's me... - I blew a gasket at Claude for ignoring ASD-STE100 and hiding behind our so‑called “house convention,” then realized the culprit is me: 700+ commits of pre-switch docs and TODOs that teach the opposite. Turning this tanker will be slow, so I’ll keep leaning on the tiller and help Claude while we keep shipping the vibe-coded Elite clone.

The Output Style, It Does Nothing! - Learning to live with Claude - I tried taming Claude’s bar-stool rambling with ASD-STE100 and even stuffed it in CLAUDE.md. It felt cleaner for a moment, but the prose stayed just as empty—while the code...