A little experiment in evading AI detection — Nick Gustafson<br>Skip to contentTLDR: I tried to evade AI detection on a heavily AI written corpus using Claude Code, 10 parallel agents, and some basic tricks. It didn't work, but it was fun.

Disclaimer: I am in no way affiliated with the people behind Pangram, but kudos to them for making something that seems to do what it says. You will have to be more clever than me to evade them.

As a learning exercise I used AI to write the series Holding the LLM Stack in Your Head. It was developed in collaboration with Claude Code and Opus 4.6/4.7/4.8. Creating it was fantastically fun, and I learned a ton. I'm going to do something like that again on a different topic, and you should too! You will learn something.

Anyway, I was pretty sure that AI detection would flag a good chunk of it, but not all of it. I had thousands of lines of back and forth with Claude, massaging the posts until they were in a state that pleased me. My assumption was that all the genuine human tokens I input into Claude Code would influence the output tokens enough to come across as human, or at least a little bit human.

I was wrong. After getting an API key to Pangram and evaluating all the prose in the 84 posts, almost every single one was flagged as 100% AI. This gave me the idea to test if I could evade that AI detection using the same AI that was used to generate that text.

First some quick stats. Thanks for the visual Claude.

84<br>series posts

~130<br>detection calls

~300k<br>words submitted

~$15<br>api credits

Round 1

My first pass at evading the AI detector wasn't very clever. I pulled down the full text of this Wikipedia page, Signs of AI Writing. I passed all those tokens into the context for each of 10 subagents, one for each arc, that were instructed to use the information in the wiki to edit each post in their arc in a way that made it sound like human writing and not AI.

Here are a few examples of things it did. Again, thanks Claude for making the visual.

Round 1 · Wikipedia rulebook · attention-from-scratch−This separation is crucial.<br>+This separation matters.

Round 1 · gpus-and-floating-point−When they do, you overflow to infinity. Your loss becomes NaN. Training crashes.<br>+When they do, you overflow to infinity, your loss becomes NaN, and training crashes.

Round 1 · sampling-strategies−The model "learned" a rich probability landscape, and greedy decoding ignores most of it.<br>+The model "learned" a whole probability distribution, and greedy decoding ignores most of it.

The edits were pretty minor, but theoretically guided by the information in the wiki. They do seem to be a bit "less AI" to me.

But yeah, as you probably guessed based on all my framing: this didn't work at all. Every post was basically still flagged as 100% AI generated.

Round 2

Round two I tried to get more clever. I gave each agent access to a script that allowed them to safely call the Pangram API on the prose of the blogs they were assigned. They were also given the wikipedia article and instructed to iterate on the content of their arc, retrying the API and seeing their AI score, until they were able to get a human label or they ran out of the credits the script assigned them.

ArcTopicPostsWords submittedfraction_ai1Math prerequisites8~27,0000.92 – 1.002Pre-transformer LMs7~20,7600.87 – 1.003Tokenization7~23,7000.87 – 1.004Transformers9~28,4780.85 – 1.005Decoding9~30,5000.87 – 1.006Inference engines9~26,6500.89 – 1.007Training10~33,9700.88 – 1.008Evaluation7~24,4000.77 – 1.009Retrieval9~29,1000.93 – 1.0010Agents9~29,5000.92 – 1.00Total 84 ~274,000 all "AI Detected"<br>Distribution of the 84: ~62 posts scored 0.99–1.00, ~15 scored 0.90–0.98, and ~7 scored 0.77–0.89. The single least-AI post was contamination at 0.77. None of them earned the label "human".

It was interesting that the AI couldn't evade the AI detection even when given multiple accesses to the AI detecting oracle. I suppose it is possible that I just didn't let it iterate enough, but I didn't wanna waste money. If I find a good open weight AI detection model I'll give it a try in a loop while I sleep.

However, if you inspect the edits, they aren't very ambitious edits. I'll address that in a sec.

Round 2 · Pangram-guided · throughput-vs-latency0.99→0.99AI−When you're running an inference service, you can't optimize for "speed." Speed is two numbers, and they fight each other.<br>+Run an inference service for a while and "speed" stops being a single thing you can chase. There are really two numbers under that word, and improving one tends to cost you the other.

Not much a difference to me.<br>You see, Claude Code was using its file editing tool to do the rewriting, which I think biased it toward keeping in place the existing structure. Sentence length, flow, structure, and other longer range patterns are probably things that the detection is keying in on.

So let's have it not use the file edit tool and instead create a whole new file from scratch...