Tokenizer Expansion: Upgrading a Model's Tokenizer in Place — Blog — Liquid AI

Connect

h2]:clear-both [&>h3]:clear-both [&_:not(pre)>code]:bg-accent-wash [&_:not(pre)>code]:text-accent [&_:not(pre)>code]:rounded [&_:not(pre)>code]:px-[0.4em] [&_:not(pre)>code]:py-[0.15em] [&_:not(pre)>code]:font-mono [&_:not(pre)>code]:text-[0.9em]">Today, we're sharing the recipe behind the new tokenizer in LFM2.5-8B-A1B . It upgrades a pre-trained model's tokenizer in place, without retraining from scratch. We doubled the vocabulary from 65K to 128K to fix the languages our original tokenizer split too finely. Hindi and Vietnamese now take roughly 2.4× and 2.6× fewer tokens, and Thai up to 4.0× fewer, which we estimate makes per-character decoding 2.2 to 3.7× faster for these languages on-device. The quality of the languages the model already handled well holds steady.<br>LFM2.5-8B-A1B is available on Hugging Face, and the expanded 128K tokenizer is released alongside it. The full method, ablations, and per-device numbers are in the technical report.<br>Why on-device tokenizers stay small<br>A tokenizer is fixed at the start of pre-training, and it splits the vocabulary in proportion to whatever the training corpus looked like then. Languages that were underrepresented at that point get broken into far more tokens per word than the ones the tokenizer was tuned for [1, 2]. Because a model runs its decoder once per output token, that extra fragmentation costs real latency, compute, and energy for users of those languages.<br>Cloud models can hide this cost behind a large vocabulary, since the embedding and output (LM-head) matrices are a small fraction of their parameters. On a small on-device model, they are not. At batch size 1, which is what we often care about on-device, decoding is limited by memory bandwidth, and the LM-head reads the whole vocabulary on every step. A larger vocabulary means a larger matrix to stream on each token and to keep in RAM, so edge models ship compact vocabularies and live with fragmentation outside their priority languages.<br>LFM2's original 65K byte-level Byte Pair Encoding (BPE) [3, 4] tokenizer was built for English, code, and a fixed set of languages, leaving little budget for Hindi, Vietnamese, or Thai. We wanted to fix that on a checkpoint we had already trained. Retraining the tokenizer and pre-training again would throw away that compute, and dropping in an unrelated third-party tokenizer would force us to solve a harder transfer problem than we needed to [5].<br>The recipe: extend, don't replace<br>Our tokenizer expansion recipe has two parts: building the new tokenizer, then adapting the model to it. Because we design the tokenizer ourselves, we can build the new one as a direct extension of the old one, which keeps everything downstream simple.

Figure 1. The tokenizer expansion pipeline. We extend the source tokenizer's BPE merges to build the 128K vocabulary (top), reinitialize the model's embeddings from it, and adapt the model in two stages before mid-training and post-training (bottom).<br>1. Continue the BPE merges<br>A BPE tokenizer is defined by an ordered list of merge rules that combine smaller tokens into larger ones. For our recipe, we seed the new tokenizer's merge table with the original merges, freeze them, and continue BPE training on a multilingual corpus. This gives us two useful properties. First, most of the original 65K tokens carry over unchanged, so their learned representations transfer directly. Second, every new token has an exact decomposition into a sequence of original tokens.<br>2. Initialize the new embeddings from what the model already knows<br>This exact decomposition makes it easy to initialize the new token’s embeddings. Tokens that carry over keep their original embedding row. Each new token gets the mean of its sub-tokens' rows. Nothing is initialized randomly, and there is no cross-tokenizer alignment to work out. Since LFM2.5-8B-A1B ties its input and output embeddings, one matrix covers both.<br>3. Adapt the model in two stages<br>Training every parameter at once degraded the parts of the model that already worked, so we split adaptation into two stages, both run before the usual mid-training and post-training:<br>Stage 1, embedding only. Train only the new rows on 600B tokens with the rest of the model frozen. The new tokens settle without disturbing the body, and this alone recovers most of the quality lost at the swap.<br>Stage 2, full continued pre-training. Unfreeze everything and continue pre-training on 400B tokens of a balanced multilingual mixture. This folds the new vocabulary into the body of the model and closes the remaining gap.<br>After stage 2, the checkpoint drops straight into the standard LFM2.5 pipeline with no other changes.<br>What the new tokenizer buys<br>The main payoff is compression where we needed it. English and code stay at parity by design, while the languages we already supported improve a little, and the languages we under-tokenized before improve a...