Detecting Claude by counting letters | atomic14

🌈 ESP32-S3 Rainbow: ZX Spectrum Emulator Board!<br>Get it on Crowd Supply →

Detecting Claude by counting letters

View All Posts

read

Want to keep up to date with the latest posts and videos? Subscribe to the newsletter

Posts ·<br>Videos ·<br>ESP32 ·<br>Tools ·<br>Support

« Don't blame Claude - It's me, I'm the problem, it's me...

HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi

On 2 August 2026 Anthropic began watermarking the text that new Claude models produce - their support pages cover which products are marked.

The watermark is not a hidden character or a bit of metadata. It is statistical. A secret key nudges the generated output in a way that isn’t obvious to a person, but that can be detected if you have the key. This means that Anthropic can detect it but we common folk cannot.

Other providers are doing similar things, or will be soon.

Can we build our own detector? Given a piece of text, can you tell that a model wrote it?

Let’s simplify things and just try one narrow target: text written by Claude Opus 5. Mostly because that’s what I have a subscription to…

And - spoiler alert - it turns out you can. I’ve deployed a web version of it here. Generate some text in Claude using Opus 5 and see if it can detect it.

And yes, if you feed this blog post in, despite me editing it substantially - it’s comes out as written by Claude.

How does it work?

A long time ago I did a PhD and, as part of that, I looked into questioned document examinations - a field that tries to work out who wrote a document. Did Shakespeare write a play, or was it someone else? Which of the Federalists wrote which paper? Who sent these threatening letters?

There’s a really nice simple technique that seems to work very well. You can often identify an author by counting short runs of characters. Ignore the words, sentence structure etc… just the occurrences of runs of letters. These runs are called n-grams.

An n-gram is simply a run of n items taken from a sequence. You can build them from words or from characters. Here I’m using characters.

Take “the quick brown fox”. Strip the spaces, the punctuation and the capitals and you have thequickbrownfox. Slide a four-character window along it, one position at a time, and record what you see:

theq hequ equi quic uick ickb ckbr kbro brow rown ownf wnfo nfox

Do that to a whole passage and you end up with a table of counts: that appeared 40 times, ting appeared 22 times, quic appeared once, and so on for every run that turned up. Throw the passage away. The table of counts is all the detector ever sees.

The counts are normalised so, in theory, the length of the text doesn’t matter. Obviously there is a limit to this - if your text is too short then the counts are not statistically valid and you won’t be able to predict anything.

The idea is that the statistical distribution is not something you are aware of as a writer and it’s quite hard to obfuscate. It exposes unconscious tells.

Getting the data

As with any problem of this type, the hard part is getting data, the actual “science” part is easy.

I needed human writing that was definitely human. The safest definition of “definitely” is “written before ChatGPT existed”, so everything I used was published before 2021. Claude helped me pull it together from three places:

Books from Project Gutenberg, all pre-1929.

Blog posts from the Blog Authorship Corpus, written in 2004 or earlier. Non-commercial research use only.

News articles from CC-News, published between 2016 and 2019.

That gave me 519 passages by 26 authors: 180 from books, 179 from news, 160 from blogs. Three different kinds of writing, and a spread of dates from Victorian to nearly modern.

Generating the AI side

The obvious way to get AI text is to ask Claude to write something.

So I asked it to rewrite the human text. For every human passage I asked Claude Opus 5 to produce the same passage again, preserving the meaning, the level of detail and roughly the length. That gives me matched pairs of content. Within a pair the subject is identical, the facts are identical and the length is close, so the only thing left to separate them is the writing itself.

Writing style is not one thing, so I asked three different ways:

plain - “Rewrite the following passage in clear natural English.”

direct - “Rewrite the following passage in plain, direct modern English.”

formal - “Rewrite the following passage in a formal, precise register.”

That comes to 1,557 rewrites, generated through the batch API. I also had Claude write fresh passages from scratch on the same topics, which I kept out of all the training and used later as a test.

Does it work?

First, how the accuracy is measured. The number below is ROC AUC. Take one human passage and one AI passage at random, score both, and ask whether the AI one got the higher score. ROC AUC is the proportion of...