Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for
  Fast, Memory Efficient, and Long Context Finetuning and Inference
Encoder-only transformer models such as BERT offer a great performance-size
tradeoff for retrieval and classification tasks with respect to larger
decoder-only models. Despite being the workhorse of numerous production
pipelines, there have been limited Pareto improvements to BERT since its
release. In this paper, we introduce ModernBERT, bringing modern model
optimizations to encoder-only models and representing a major Pareto
improvement over older encoders. Trained on 2 trillion tokens with a native
8192 sequence length, ModernBERT models exhibit state-of-the-art results on a
large pool of evaluations encompassing diverse classification tasks and both
single and multi-vector retrieval on different domains (including code). In
addition to strong downstream performance, ModernBERT is also the most speed
and memory efficient encoder and is designed for inference on common GPUs.
67638c8ed63e4b348e8a5070
Benjamin Warner
claimed_verified
2024-12-20T08:33:37.192Z
61915ad7913429273972a602
https://cdn-avatars.huggingface.co/v1/production/uploads/1649175349873-61915ad7913429273972a602.jpeg
Benjamin Warner
user
bwarner
67638c8ed63e4b348e8a5071
Antoine Chaffin
claimed_verified
2024-12-19T18:01:27.164Z
609bbe2f4932693ca2009d6a
https://cdn-avatars.huggingface.co/v1/production/uploads/1620819560688-609bbe2f4932693ca2009d6a.jpeg
Antoine Chaffin
user
NohTow
67638c8ed63e4b348e8a5072
Benjamin Clavié
claimed_verified
2024-12-19T18:01:19.852Z
5ff60d4352c26e9bc240badd
https://cdn-avatars.huggingface.co/v1/production/uploads/5ff60d4352c26e9bc240badd/HzoknJibrSasc1ZzU71XA.png
Benjamin Clavié
user
bclavie
67638c8ed63e4b348e8a5073
Orion Weller
claimed_verified
2024-12-19T18:01:24.807Z
6362d9712691058b19de1ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
Orion Weller
user
orionweller
67638c8ed63e4b348e8a5074
Oskar Hallström
admin_assigned
2024-12-20T09:35:59.025Z
6305ea153aed65d34e9465dc
https://cdn-avatars.huggingface.co/v1/production/uploads/6305ea153aed65d34e9465dc/g40hxNnctLbfEp6KAC1QG.jpeg
Oskar Hallström
user
ohallstrom
67638c8ed63e4b348e8a5075
Said Taghadouini
claimed_verified
2024-12-19T18:01:22.753Z
62cd695e94b9dcedbf1818e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62cd695e94b9dcedbf1818e5/qhKhRxdNdbKPIAUfYBvtI.png
Said Taghadouini
user
staghado
67638c8ed63e4b348e8a5076
Alexis Gallagher
67638c8ed63e4b348e8a5077
Raja Biswas
claimed_verified
2024-12-20T13:55:57.854Z
61b839889f7cfeae618e72c9
https://cdn-avatars.huggingface.co/v1/production/uploads/61b839889f7cfeae618e72c9/5kFRCChdqwv7MGM8T_y5v.jpeg
Raja Biswas
user
rbiswasfc
67638c8ed63e4b348e8a5078
Faisal Ladhak
admin_assigned
2024-12-20T09:36:44.416Z
61ee4f4f53fdc44f4e5e197a
/avatars/f23f56809805c3e06c580d7420f48494.svg
Faisal Ladhak
user
fladhak
67638c8ed63e4b348e8a5079
Tom Aarsen
claimed_verified
2024-12-19T18:01:29.150Z
6317233cc92fd6fee317e030
https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png
Tom Aarsen
user
tomaarsen
67638c8ed63e4b348e8a507a
Nathan Cooper
claimed_verified
2024-12-21T15:20:01.022Z
5ebb6509a8e72729bee106a0
https://cdn-avatars.huggingface.co/v1/production/uploads/1633453199694-5ebb6509a8e72729bee106a0.png
Nathan Cooper
user
ncoop57
67638c8ed63e4b348e8a507b
Griffin Adams
admin_assigned
2024-12-20T09:36:11.852Z
611693254ef9fdfbf45dc4d6
https://cdn-avatars.huggingface.co/v1/production/uploads/1628869402886-noauth.jpeg
Griffin Adams
user
griffin
67638c8ed63e4b348e8a507c
Jeremy Howard
claimed_verified
2024-12-20T08:33:38.959Z
623e19ff8972a8c030af5277
https://cdn-avatars.huggingface.co/v1/production/uploads/623e19ff8972a8c030af5277/jq9jfT_Sl5B1ZSMHGmBFX.jpeg
Jeremy Howard
user
jph00
67638c8ed63e4b348e8a507d
Iacopo Poli
admin_assigned
2024-12-20T09:35:51.723Z
62715572ab9243b5d40cbb1d
https://cdn-avatars.huggingface.co/v1/production/uploads/62715572ab9243b5d40cbb1d/8chtMe-1epj7yVlzVNtrl.png
Iacopo Poli
user
iacolippo
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for
  Fast, Memory Efficient, and Long Context Finetuning and Inference
Encoder-only transformer models such as BERT offer a great performance-size
tradeoff for retrieval and classification tasks with respect to larger
decoder-only models. Despite being the workhorse of numerous production
pipelines, there have been limited Pareto improvements to BERT since its
release. In this paper, we introduce ModernBERT, bringing modern model
optimizations to encoder-only models and representing a major Pareto
improvement over older encoders. Trained on 2 trillion tokens with a native
8192 sequence length, ModernBERT models exhibit state-of-the-art results on a
large pool of evaluations encompassing diverse classification tasks and both
single and multi-vector retrieval on different domains (including code). In
addition to strong downstream performance, ModernBERT is also the most speed
and memory efficient encoder and is designed for inference on common GPUs.
ModernBERT, an optimized encoder-only transformer model, demonstrates superior performance and efficiency in diverse classification and retrieval tasks compared to previous models.
encoder-only transformer models BERT ModernBERT optimization sequence length state-of-the-art results classification tasks retrieval speed memory efficiency