Reconstructing Claude's tokenizer

tokenize.rs

Reconstructing Claude's tokenizer

ctok on GitHub ↗<br>Read the write-up ↗

Claude publishes no tokenizer. count_tokens hands back a<br>number, one round trip at a time, and nothing about where it came<br>from. ctok returns the same number offline, and the pieces behind it.

Our best estimate of the vocabularies: 49,152 entries in v3, a third<br>of that in v4.7 at 16,384, with some of both reserved for image<br>tokens.

Markers ride inside the piece they belong to: word edges, capitals,<br>and raw bytes where the vocabulary has no piece for a character.<br>The write-up goes through them.

The cost of a small vocabulary

Claude 5's tokenizer comes at a cost, but less than its size suggests.<br>Its vocabulary is about 12&times; smaller than o200k, while its token<br>bill is 1.24&times; to 2.44&times; larger on these samples. Common pieces<br>do most of the compression; a larger vocabulary spends the rest on<br>rarer words and scripts.

Claude Opus 5 and GPT-5.6 Sol both charge $5 per million input<br>tokens, so the token ratio is the cost ratio.

ctok on GitHub ↗<br>On the Biology of Claude's Tokenizer ↗