Marcel R酶d on X: "Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 馃У https://t.co/zxzXPFBCHv" / X<br>Post
Log inSign up
Post
Marcel R酶d<br>@marcelroed
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 馃У
span:not(:empty)~span:not(:empty)]:before:content-['路'] [&>span:not(:empty)~span:not(:empty)]:before:px-1 [&>span:not(:empty)~span:not(:empty)]:before:shrink-0">6:58 PM 路 Jul 21, 202625.7KViews
808<br>44044<br>1990199<br>1150115
Read 8 replies
New to X?<br>Sign up now to get your own personalized timeline!<br>Sign up with GoogleSign up with AppleCreate account<br>By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.
Relevant people<br>Marcel R酶d@marcelroedFollow
Don't miss what's happening<br>People on X are the first to know.
Log inSign up
Trending now