How to Produce a Pangram 4 False Positive
Freddie deBoer
SubscribeSign in
How to Produce a Pangram 4 False Positive<br>they're impossible to 100% eliminate, but that's OK if we're chill and have integrity
Freddie deBoer<br>Aug 03, 2026
164
19<br>19
Share
I was unsurprised to find that my post on the AI writing detector Pangram was controversial. After all, people who argue for a living have suddenly been caught up in a lot of angst about all of this and people who do discourse for a living respond to discourse about discourse with discourse. And when Substack quoted a different piece of mine when they announced their Pangram-powered automated AI checker, I was inevitably thrust into the broader conversation on these things.<br>Funnily enough, since Substack’s partnership with Pangram was announced I have been repeatedly invoked as both a supporter and a critic of Pangram and AI writing detection, depending on who’s invoking. (This is nice because it flatters my self-conception as a very special snowflake.) The truth is that I’m neither all booster or all critic; I think these technologies are both inevitable and necessary, and I think there’s no reason for them to become a problem if people understand that they are fundamentally and permanently limited. Of course, the difficulty is that some people don’t understand that. In a deeper sense, I’m intellectually engaged by this stuff. I would say I’m ultimately someone who has a general idea of how these technologies work, though not a particularly sophisticated one, and someone who’s not threatened by the accusation of using LLMs, and someone who likes to tinker. I will continue to give you this advice about this topic: don’t panic. Think! Engage. Play with the various products. Experiment. Iterate. You know if you’re guilty of passing of LLM writing as your own or not. If you are, that’s dishonest and you should stop. If you use LLMs to edit or enhance your writing, be transparent about it. If your stuff is all LLM-produced and you feel embarrassed about it, ask yourself why you feel embarrassed. Interrogate yourself. But whatever you do, think. So many people refusing to engage for fear of being seen as protesting too much doesn’t help anyone. Think. Talk. Play.<br>I do want to make a couple of points. The first is both simple and philosophical: AI detection software is AI. LLM detection depends on technologies that are fundamentally an expression of LLM principles. If you’re a true-blue AI hater, it’s sort of weird to be a loud AI detection software partisan for that reason. You will have to sort out the ethics of this scenario on your own.<br>The second point is this. You have a lot of people who have justifiable concerns over the use of this technology. You have a lot of people who embrace the technology but who have a good sense of the limitations and an understanding that we need to be careful and fair in their use. But you also have this set of people who seem to me to have a bizarre level of faith that false positives are genuinely not possible. Indeed, the part of my post that has proven to have attracted the most criticism is people who complain that I said that I could produce false positives. That this is impossible - despite Pangram itself saying that it is very much possible! - is an article of faith among some. (Go look on Reddit and you can see for yourself.) I think this is screwy and you only need to spend a few minutes thinking about it to figure out why.<br>What does an LLM writing detector do? Obviously, an LLM writing detector looks for the textual features of LLM writing - that is, for the placement of characters and spaces that are common in LLM writing, for patterns in the use of words, punctuation, and spacing that are consistently found in text produced by LLMs. But where do those patterns come from? They come from human written texts that are in the training corpora that are used to build LLMs. Every LLM textual pattern - every LLM textual pattern - is ultimately the product of human text production. The very nature of an LLM is to model human text based on human text. BThat LLMs reconfigure and recombine human texts in predictable ways is why systems like Pangram can exist. But it was all human once. LLMs do not invent, cannot invent, can only remixing, reassemble what already exists, though often into an order no one has quite used before. LLMs reconstitute the human. That is both why LLM detection is possible and why LLM detection will always have a false positive issue; it’s the dual nature of the underlying technology. None of this should be controversial.<br>I’m impressed by Pangram’s advertised 1-in-10,000 error rate because what they’re attempting is genuinely very hard to do: find consistent non-human textual patterns in texts that might have been produced by digital systems that by their very nature ingest and recombine text patterns that were human before the were digital. Right? The LLM companies are investing monstrous amounts of...