GitHub - sanderland/ctok: Claude's Tokenizer, Offline, Kinda · GitHub
/" data-turbo-transient="true" />
Skip to content
Search/
Sign in<br>Sign upAppearance settings
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
sanderland
ctok
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star<br>54
main
BranchesTags
Go to file
CodeOpen more actions menu
Folders and files<br>NameNameLast commit message<br>Last commit date<br>Latest commit
History<br>18 Commits<br>18 Commits
.github/workflows
.github/workflows
ctok
ctok
tests
tests
.gitignore
.gitignore
LICENSE
LICENSE
README.md
README.md
pyproject.toml
pyproject.toml
uv.lock
uv.lock
View all files
Repository files navigation
ctok
ctok reconstructs Claude token counts offline, with no API call, network access, or runtime<br>dependencies. It is unofficial and is not affiliated with Anthropic.
The reconstruction targets counts. Claude does not expose token boundaries, so tokenize() returns<br>one valid minimum-cost tiling, not a claim about Anthropic's exact segmentation. The research behind<br>the model is described in On the biology of Claude's tokenizer.
Quick start
from ctok import token_count, tokenize
token_count("hello, world") # 10, using the v3 family<br>token_count("hello, world", "4.7") # 15<br>token_count("hello, world", "5.0") # 10
tokens = tokenize("NASA likes tokenizers")<br>assert len(tokens) == token_count("NASA likes tokenizers")
The command-line interface prints the marked stream and its tiling:
ctok "hello, world"
Supported families
version is always a string, e.g. "4.7", compared component by component — "4.10" sorts after<br>"4.9", not below "4.2". A float can't make that distinction (Python collapses the literal<br>4.10 to 4.1 before any code here sees it), so a non-str version raises TypeError.
requested version<br>family<br>model generation
"3.0"<br>v3, the default<br>Claude 3 through Opus 4.6
"4.7"<br>v4.7<br>Opus 4.7 through 4.9
version >= "5.0"<br>v5<br>Opus 5 and Sonnet 5
v5 is v4.7 with a slightly different fixed overhead.
How it works
For one user message, ctok:
normalizes the text, including NFC and family-specific quote folding;
rewrites it into a stream with word, case, and byte markers;
finds a minimum-cost tiling over the measured vocabulary and UTF-8 byte fallback;
adds the measured message frame.
token_count(text) is len(tokenize(text)). The output notation makes internal structure visible:
notation<br>meaning
⟨bow⟩, ⟨eow⟩<br>word boundaries
⟨shift⟩, ⟨caps⟩<br>case rewrites
⟨0xNN⟩<br>a byte-fallback token
⟨pad⟩<br>part of the single-message frame
Measured accuracy
These results compare ctok with recorded count_tokens responses:
corpus<br>role<br>v3 exact<br>v4.7 exact
Goldfish, 350 languages and 350,000 rows<br>mining<br>350,000<br>350,000
MultiPL-E, 22 programming languages<br>held out<br>22<br>22
Rosetta Code, 1,741 documents<br>mining<br>1,741<br>1,741
Rosetta Code, separate 250 documents<br>mining<br>250<br>250
UDHR, 501 languages<br>mining (in-sample since 2026-08-12)<br>501<br>501
v5 has the same content result as v4.7 because it uses the same vocabulary.
The stored measurement sets contain no under-counts: 0 of 1,664,940 v3 texts and 0 of 1,722,961<br>v4.7 texts. This is an empirical result, not a guarantee for arbitrary input. Goldfish, Rosetta and<br>(since its final six pieces were selected by bisecting it) UDHR may select candidates; MultiPL-E<br>never does, and is the one remaining held-out corpus in this table.
Run the public gates with:
uv run pytest<br>uv run python tests/gates.py --markdown
Vocabulary evidence
The two vocabulary files contain 48,645 v3 pieces and 15,240 v4.7 pieces. Every entry has a fixed<br>membership witness or is one of the structural marker atoms checked by the test suite.
from ctok import pieces, witness
len(pieces("4.7")) # 15240<br>witness("⟨bow⟩the⟨eow⟩", "4.7")<br># {'probe': 'the', 'raw': 12, 'kind': 'raw'}
A witness says that one marked piece costs one token in a calibrated probe. It does not prove the<br>encoder rewrite or resolve ties between equal-cost tilings. tests/test_witness.py checks every<br>published witness and requires complete witnessed-or-special coverage.
License
MIT
About<br>Claude's Tokenizer, Offline, Kinda<br>Resources<br>Readme<br>MIT license<br>Activity<br>Stars<br>54 stars<br>Watchers<br>0 watching<br>Forks<br>3 forks<br>Report repository
Releases
Packages
Contributors
Languages
You can’t perform that action at this time.