Finally, a Replacement for BERT: Introducing ModernBERT – Answer.AI

Finally, a Replacement for BERT

Note

This is a cross-post of the announcement blog post posted on the 🤗 HuggingFace blog.

TL;DR

This blog post introduces ModernBERT, a family of state-of-the-art encoder-only models representing improvements over older generation encoders across the board, with a 8192 sequence length, better downstream performance and much faster processing.

ModernBERT is available as a slot-in replacement for any BERT-like models, with both a base (149M params) and large (395M params) model size.

Click to see how to use these models with transformers

ModernBERT will be included in v4.48.0 of transformers. Until then, it requires installing transformers from main:

pip install git+https://github.com/huggingface/transformers.git

Since ModernBERT is a Masked Language Model (MLM), you can use the fill-mask pipeline or load it via AutoModelForMaskedLM. To use ModernBERT for downstream tasks like classification, retrieval, or QA, fine-tune it following standard BERT fine-tuning recipes. ⚠️ If your GPU supports it, we recommend using ModernBERT with Flash Attention 2 to reach the highest efficiency. To do so, install Flash Attention as follows, then use the model as normal:

pip install flash-attn

Using AutoModelForMaskedLM:

from transformers import AutoTokenizer, AutoModelForMaskedLM<br>model_id = "answerdotai/ModernBERT-base"<br>tokenizer = AutoTokenizer.from_pretrained(model_id)<br>model = AutoModelForMaskedLM.from_pretrained(model_id)<br>text = "The capital of France is [MASK]."<br>inputs = tokenizer(text, return_tensors="pt")<br>outputs = model(**inputs)<br># To get predictions for the mask:<br>masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)<br>predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)<br>predicted_token = tokenizer.decode(predicted_token_id)<br>print("Predicted token:", predicted_token)<br># Predicted token: Paris

Using a pipeline:

import torch<br>from transformers import pipeline<br>from pprint import pprint<br>pipe = pipeline(<br>"fill-mask",<br>model="answerdotai/ModernBERT-base",<br>torch_dtype=torch.bfloat16,<br>input_text = "He walked to the [MASK]."<br>results = pipe(input_text)<br>pprint(results)

Note: ModernBERT does not use token type IDs, unlike some earlier BERT models. Most downstream usage is identical to standard BERT models on the Hugging Face Hub, except you can omit the token_type_ids parameter.

Introduction

BERT was released in 2018 (millennia ago in AI-years!) and yet it’s still widely used today: in fact, it’s currently the second most downloaded model on the HuggingFace hub, with more than 68 million monthly downloads, only second to another encoder model fine-tuned for retrieval. That’s because its encoder-only architecture makes it ideal for the kinds of real-world problems that come up every day, like retrieval (such as for RAG), classification (such as content moderation), and entity extraction (such as for privacy and regulatory compliance).

Finally, 6 years later, we have a replacement! Today, we at Answer.AI and LightOn (and friends!) are releasing ModernBERT. ModernBERT is a new model series that is a Pareto improvement over BERT and its younger siblings across both speed and accuracy . This model takes dozens of advances from recent years of work on large language models (LLMs), and applies them to a BERT-style model, including updates to the architecture and the training process.

We expect to see ModernBERT become the new standard in the numerous applications where encoder-only models are now deployed, such as in RAG pipelines (Retrieval Augmented Generation) and recommendation systems.

In addition to being faster and more accurate, ModernBERT also increases context length to 8k tokens (compared to just 512 for most encoders), and is the first encoder-only model that includes a large amount of code in its training data. These features open up new application areas that were previously inaccessible through open models, such as large-scale code search, new IDE features, and new types of retrieval pipelines based on full document retrieval rather than small chunks.

But in order to explain just what we did, let’s first take a step back and look at where we’ve come from.

Decoder-only models

The recent high-profile advances in LLMs have been in models like GPT, Llama, and Claude. These are decoder-only models, or generative models. Their ability to generate human-like content has enabled astonishing new GenAI application areas like generated art and interactive chat. These striking applications have attracted major investment, funded booming research, and led to rapid technical advances. What we’ve done, essentially, is port these advances back to an encoder-only model.

Why? Because many practical applications need a model that’s lean and mean ! And it doesn’t need to be a generative model.

More bluntly, decoder-only models are too big, slow, private , and expensive for many...