LFM2.5-Encoders: Fast at Long Context, Even on CPU — Blog — Liquid AI

Connect

h2]:clear-both [&>h3]:clear-both [&_:not(pre)>code]:bg-accent-wash [&_:not(pre)>code]:text-accent [&_:not(pre)>code]:rounded [&_:not(pre)>code]:px-[0.4em] [&_:not(pre)>code]:py-[0.15em] [&_:not(pre)>code]:font-mono [&_:not(pre)>code]:text-[0.9em]">Today, we release two new encoder models: LFM2.5-Encoder-230M and LFM2.5-Encoder-350M . Both are bidirectional encoders built on the LFM2 hybrid architecture. They are designed to be fine-tuned for classification, natural language understanding, and token-level tasks. On these, they match the quality of larger encoders while scaling much more gently with input length up to a context length of 8,192 tokens. This keeps document-scale workloads fast on the hardware you already have, including CPU-only environments.<br>The LFM2.5-Encoder-230M and LFM2.5-Encoder-350M models are available today on Hugging Face. Check out our docs on how to run and fine-tune them locally.<br>Why a general-purpose encoder?<br>Last month, we released LFM2.5-Retrievers, a pair of models built for multilingual retrieval tasks. LFM2.5-Encoders are close relatives from the same LFM family, but they are built for a broader purpose. The earlier LFM2.5-Retrievers were trained for retrieval tasks, while the LFM2.5-Encoders are pre-trained with a masked-language objective, so they can be adapted to a range of downstream tasks, including classification, token-level tasks, and retrieval. Because retrieval is only a subset of what encoders enable, we chose to build a general-purpose encoder rather than adapt the existing retrievers to a new task.<br>Classifiers, intent routers, and safety filters are all built on encoders. They run constantly, often on CPUs rather than GPUs, and increasingly on longer inputs. Originally, BERT [1] established this pattern, and more recent releases such as ModernBERT [2] have shown how much potential there is to improve accuracy, speed, and context length. LFM2.5-Encoders continue that line of work on the LFM2 architecture, whose cost grows slowly as inputs get longer. The result is document-scale understanding that stays fast even without a GPU.<br>Architecture<br>The encoders build on the LFM2 hybrid backbone and are initialized from LFM2.5-230M and LFM2.5-350M, respectively. We adapt this causal decoder into a bidirectional encoder with a small set of changes:<br>We replace the causal attention mask with a bidirectional one , so every token attends to both its left and right context.<br>We make the LFM2 short convolutions non-causal by using symmetric center padding, so they mix local information around each token. In practice, this means each token's short convolution now reads from its neighbors on both sides, rather than only the tokens that came before it.<br>We train with a masked language modeling objective , masking 30% of tokens. That is denser than BERT's 15%, following evidence that a higher mask rate helps at this scale [3].

Both models are trained in a two-phase process : Phase 1 establishes general language competence with a short-context masked-language objective on a large, packed web corpus, at a 1,024-token context. Phase 2 is a long-context adaptation phase, extending the context to 8,192 tokens on the full data mix, which strengthens the encoder's factual, legal, and multilingual competence.<br>Benchmarks<br>For each benchmark task, we run a full supervised fine-tune and report that fine-tuned model's score. The results below span 14 models across 17 tasks from GLUE, SuperGLUE, and multilingual classification tasks.<br>We select the learning rate per task on held-out seeds, then report the mean across five fresh seeds that never participated in selection, so the numbers are stable from run to run. The full framework, per-task launchers, and raw results are open-sourced.

LFM2.5-Encoder-350M ranks fourth of the 14 models, behind only three larger ones, including a 3.5B model nearly 10 times its size. LFM2.5-Encoder-230M beats ModernBERT-base and every EuroBERT [4] while being smaller than most of them. Both LFM2.5-Encoders also score well above our own LFM2.5-Retrievers on these tasks, which is why we built a general-purpose encoder rather than reusing the retrievers.<br>Inference<br>The LFM2 backbone was built for fast inference, which our encoders inherit. For an encoder, speed comes down to how quickly it turns an input into a result, which depends on the input length. Since LFM2.5-Encoders and ModernBERT both natively support an 8,192-token context, we can measure that speed across the full supported range.

Our encoders show their biggest edge on CPU. Here, LFM2.5-Encoder-230M is the fastest model from 1K tokens up, and the gap widens with length. Both ModernBERT models’ throughput drops steeply with length, while our LFM2.5-Encoders climb to a peak in the mid-range before tapering, so they pull ahead and stay there. The gap widens with increasing input lengths: At 8,192 tokens, ModernBERT-base takes over...