mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Encoder-only languages models are frequently used for a variety of standard
machine learning tasks, including classification and retrieval. However, there
has been a lack of recent research for encoder models, especially with respect
to multilingual models. We introduce mmBERT, an encoder-only language model
pretrained on 3T tokens of multilingual text in over 1800 languages. To build
mmBERT we introduce several novel elements, including an inverse mask ratio
schedule and an inverse temperature sampling ratio. We add over 1700
low-resource languages to the data mix only during the decay phase, showing
that it boosts performance dramatically and maximizes the gains from the
relatively small amount of training data. Despite only including these
low-resource languages in the short decay phase we achieve similar
classification performance to models like OpenAI's o3 and Google's Gemini 2.5
Pro. Overall, we show that mmBERT significantly outperforms the previous
generation of models on classification and retrieval tasks -- on both high and
low-resource languages.
68bf9d4b207285de11b07bda
Marc Marone
68bf9d4b207285de11b07bdb
Orion Weller
claimed_verified
2025-09-12T16:20:18.815Z
6362d9712691058b19de1ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
Orion Weller
user
orionweller
68bf9d4b207285de11b07bdc
William Fleshman
68bf9d4b207285de11b07bdd
Eugene Yang
68bf9d4b207285de11b07bde
Dawn Lawrie
68bf9d4b207285de11b07bdf
Benjamin Van Durme
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Encoder-only languages models are frequently used for a variety of standard
machine learning tasks, including classification and retrieval. However, there
has been a lack of recent research for encoder models, especially with respect
to multilingual models. We introduce mmBERT, an encoder-only language model
pretrained on 3T tokens of multilingual text in over 1800 languages. To build
mmBERT we introduce several novel elements, including an inverse mask ratio
schedule and an inverse temperature sampling ratio. We add over 1700
low-resource languages to the data mix only during the decay phase, showing
that it boosts performance dramatically and maximizes the gains from the
relatively small amount of training data. Despite only including these
low-resource languages in the short decay phase we achieve similar
classification performance to models like OpenAI's o3 and Google's Gemini 2.5
Pro. Overall, we show that mmBERT significantly outperforms the previous
generation of models on classification and retrieval tasks -- on both high and
low-resource languages.
mmBERT, an encoder-only language model pretrained on multilingual text, achieves high performance on classification and retrieval tasks using an inverse mask ratio schedule and inverse temperature sampling ratio, particularly benefiting from the inclusion of low-resource languages.
encoder-only language model mmBERT inverse mask ratio schedule inverse temperature sampling ratio multilingual text low-resource languages classification retrieval OpenAI's o3 Google's Gemini 2.5 Pro