The Biology of Claude's Tokenizer: Reverse Engineered

On the Biology of Claude's Tokenizer - by Sander Land

Token Contributions

SubscribeSign in

On the Biology of Claude's Tokenizer Explaining how Claude's mysterious tokenizer seems to work

Sander Land Jul 29, 2026

Claude's tokenizer remains a mystery, and with the recent 40%+ increase in token cost for basic English text, even more so. While everyone else ships a fairly standard byte-level BPE tokenizer, Anthropic seems to have placed their bets on something unique. In this post I'll go through the most interesting properties I've found while reconstructing Claude's tokenizer. Code is available here: https://github.com/sanderland/ctok Thanks for reading Token Contributions! Subscribe for free to receive new posts and support my work.

Claude’s tokenizer is not BPE

A pairwise BPE construction requires every non-base token to be built from two smaller vocabulary pieces, so a one-token span must have at least one internal split into two things that are themselves tokens. That’s testable. The Hangul syllable 최 (choi) is the first example I found. It is one token (which makes sense, a very common surname), but testing other Korean syllables which share a common UTF-8 prefix/suffix shows the 2-byte prefix and suffix are not tokens in any other character (i.e. all characters sharing a prefix or suffix cost 3, their length in bytes). So what is it? The one rule that seems to consistently explain the token counts is a minimum-piece tokenization as in MinGram or PathPiece. Vocabulary size

My best estimate of vocabulary size is around 49-55k for Claude v3-v4.6 and around 16-20k after, based on the list of tokens found so far. Boundary markers and spaces

Boundary markers on words

One of the main novelties of the Claude tokenizer is wrapping word-like spans in boundary markers. I use ^ and $ as notation for these inferred begin- and end-of-word tokens. For example, “tokenizers” becomes [^token][izers$] For example “semiconduct”, “usercontent” are two tokens and “romagnet” is three tokens in isolation, but the tokens here are [^semiconduct][$], [^][usercontent$] and [^][romagnet][$], that is, the phrases are tokens but just in the right context. And indeed, “semiconductromagnetusercontent” tokenizes to the expected three tokens. Similarly, “telecommunications” is a single token, but “telecommunicationsy” is four ([^telecommun][ic][ation][sy$]) This also clarifies the conclusion of my previous post, where I suggested that most Claude tokens were whole words. What was really being measured was whether the fully marked form (^word$) was a one-token vocabulary entry. Boundary markers on punctuation

Spans of punctuation characters also get markers, but only on the side where they border a space, e.g. the punctuation in a== b becomes ==$ Boundary markers eat spaces

At first sight, these markers may seem a little wasteful of vocabulary space, but this is compensated for by not needing different variants for word-with-a-space and word-without-a-space. Specifically, I think that any space in $^ is removed before encoding, and $^ becomes a space during decoding. This means that many whitespaces in code and normal text are “free”, and most of all, the model doesn’t split its training data between “space prefixed” and “non space” variants. CapsCode

Another relative novelty is how capitalization is handled. Much like space+word being a separate token feels redundant, so does having duplicate tokens for nasa, Nasa, and NASA. For title-case and all-caps, Claude appears to use a shift marker or caps marker and then a lowercased form: Token → ↑ token NASA → ⇪ nasaThe marker itself can be part of a token, or be separate (just like ^ and $). Notably, it only applies to a whole pretoken, not for every capital letter. Thus: Token → marked ↑ token NASA → marked ⇪ nasa GaN → literal GaN WiFi → literal WiFiThus, while MERCHANTABILITY is a single token [⇪^merchantability$], adding a lowercase letter “q” breaks both the caps-code and word marking, turning it into [^M][ER][C][H][A][N][T][A][B][I][L][I][T][Y][q$]. The newer v4.7+ tokenizer has simplified this a lot, and removes the caps-lock code. Whitespace

There are tokens for spans of spaces, newlines and tabs. Weirdly, 1,2,…,30,32,33,40,48,64,98,128 newlines are tokens, but 31 is not. Message overhead

There appears to be a fixed overhead of 7-8 tokens, such that “a” costs 8, but a single digit cost 9 tokens. However, closer investigation suggests this is due to the surrounding chat formatting beginning with ^ and ending in \n\n (along with some other unknown content). In particular the ^ start can really lead to some puzzling results when trying to find out what is a token and what is not. Odds and Ends

Han, Korean, emoji and many rare scripts do not have word markers, and are always tokenized character-by-character.

For characters that don’t have a dedicated token, UTF-8 byte fallback is used, but only within a character, and strictly prefix-based (e.g....

The Biology of Claude's Tokenizer: Reverse Engineered

Related Articles

(no title)

Scientists reverse brain aging, with a nasal spray

NHS apologises and admits Palantir have access to identifiable patient data

AI has torched the market for junior programmers

Al Vigier: Canada's AI strategy shouldn't include secret Palantir bills