Skip to content
Search

ModernBERT

lastmod 2024-12-20created 2024-12-201 edits

ModernBERT is a new encoder-only model that applies the latest techniques, improving on BERT and its successors in speed and accuracy. It supports a long context length of 8192 tokens and was trained on data that includes code. It can be used in a wide range of applications and is especially suited to large-scale code search and new IDE features

Introduction

  • BERT was released in 2018 and is still widely used, being especially suited to retrieval, classification, and entity extraction.
  • ModernBERT is a replacement for BERT that achieves a Pareto improvement in speed and accuracy.
  • Its long context length and inclusion of code data open up new application areas.

Decoder-only models

  • Decoder-only models such as GPT, Llama, and Claude are generative models that can produce human-like content.
  • However, these models are large, slow, and costly.
  • Encoder-only models are practical and efficient, and suit many tasks.

Encoder-only models

  • Encoder-only models represent the input by converting it into numeric vectors.
  • Decoder-only models cannot see future tokens, but encoder-only models can see tokens in both directions, which makes them efficient.
  • Encoder-only models are used in many applications and are especially important in RAG pipelines and recommendation systems.

Performance overview

  • ModernBERT shows high accuracy across many tasks, and is faster and uses less memory than DeBERTaV3.
  • Up to 3x faster than other high-quality models in long-context inference.
  • Shows unmatched performance in code search, opening up possibilities for developing new applications.

Efficiency

  • ModernBERT values practicality and performs fast across a variety of input lengths.
  • 2-3x faster than other models on long-context inputs.
  • Can use larger batch sizes, so it can be used effectively even on small GPUs.

What makes ModernBERT modern

  • ModernBERT achieves its improvements by applying modern engineering to an encoder model.
  • Adopts the Transformer++ architecture to improve performance.
  • Emphasizes efficiency and modern data scale and sources.

A new Transformer

  • ModernBERT adopts the Transformer++ architecture to improve performance.
  • Uses RoPE to improve positional encoding, and uses GeGLU layers to replace the MLP layers.
  • Removes unnecessary bias terms to optimize parameter usage.

Upgrading a Honda Civic for the race track

  • ModernBERT puts weight on speed and can be used efficiently in a wide range of applications.
  • Takes advantage of the speed improvements of Flash Attention 2 to increase efficiency.
  • Reduces wasted computation through Alternating Attention, Unpadding, and Sequence Packing.

Paying attention to hardware

  • ModernBERT emphasizes hardware design and is designed to deliver optimal performance on a variety of GPUs.
  • The model was designed with a deep and narrow architecture and hardware efficiency in mind.

Training

  • ModernBERT is trained on data from a variety of sources, using 2 trillion tokens.
  • Through a three-stage training process, it shows excellent performance across many tasks.
  • Uses batch size warmup in the early training stage to speed things up.

Conclusion

  • ModernBERT improves the performance of encoder-only models by applying the latest techniques.
  • It delivers strong performance across many tasks, with an attractive size/performance ratio.
  • Looking forward to creative uses from the community, a contest for demos is under way.

Hacker News comments

  • A Replacement for BERT | Hacker News
  • Jeremy from Answer.AI hopes the new model release can become the foundation for many startups and projects
    • What is mentioned in the blog post is only the tip of the iceberg, and there are many opportunities to fine-tune the model in many ways
  • Encoder-only models get more than 1 billion downloads a month, three times more than decoder-only models
    • This is partly because decoder users use API calls instead of Hugging Face, and partly because encoders are the unsung heroes of most serious ML applications
    • Ranking, recommendation, RAG, and the like need an encoder, and models from the BERT, RoBERTa, and ALBERT families are typically used
  • A few years ago, using BERT models for things like summarization felt like a miracle
    • Plans to wait until Ollama adds it to its library; the recent speedups in LLMs are impressive
    • Apple has supported BERT models in its developer SDK, and wonders how quickly it will update to the new techniques
  • Read the paper and was impressed by the addition of local attention layers
    • Experimented with them for years in the Lucidrains repositories, and was surprised they had not progressed further
    • Inference speed is great, and there are many improvements such as removing NSP, increasing masking, RoPE, and long context
    • Would like to build a "ModernTinyBERT", but it is hard because the layers are intricately intertwined
  • Wonders where BERT models are used today
    • Understood them to be a better alternative to LLMs for certain tasks, since they can understand context better in both directions
    • But LLMs are also powerful, so the difference may be small
  • Wonders whether the model can be fine-tuned with SentenceTransformers
    • ColBERT is included in the benchmarks; wonders whether answerai-colbert-small-v2 is coming soon
  • Wonders whether anyone has seen RAG evaluations of ModernBERT
  • The Answer.ai team is delivering today; praises Jeremy and the team for a job well done
  • Wonders whether the model supports only English, and whether there are plans to publish a multilingual model or monolingual models for other languages
  • Not naming the model ERNIE was a missed opportunity

See also

Favorite site

author Zer0 sitemap