EuroBERT: Scaling Multilingual Encoders for European Languages
General-purpose multilingual vector representations, used in retrieval,
regression and classification, are traditionally obtained from bidirectional
encoder models. Despite their wide applicability, encoders have been recently
overshadowed by advances in generative decoder-only models. However, many
innovations driving this progress are not inherently tied to decoders. In this
paper, we revisit the development of multilingual encoders through the lens of
these advances, and introduce EuroBERT, a family of multilingual encoders
covering European and widely spoken global languages. Our models outperform
existing alternatives across a diverse range of tasks, spanning multilingual
capabilities, mathematics, and coding, and natively supporting sequences of up
to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering
insights into our dataset composition and training pipeline. We publicly
release the EuroBERT models, including intermediate training checkpoints,
together with our training framework.
67ce9626e5cdfda52b9e8839
Nicolas Boizard
admin_assigned
2025-03-10T09:42:06.860Z
62be186a5f59ff2320e6e32b
https://cdn-avatars.huggingface.co/v1/production/uploads/62be186a5f59ff2320e6e32b/W_emoC2uItM-MJZyCfIKI.png
Nicolas-BZRD
user
Nicolas-BZRD
67ce9626e5cdfda52b9e883a
Hippolyte Gisserot-Boukhlef
admin_assigned
2025-03-10T09:42:13.176Z
65fa95405355a52c784633fc
https://cdn-avatars.huggingface.co/v1/production/uploads/65fa95405355a52c784633fc/rSfBUHPa7eSAsLd8DuOq4.png
Hippolyte Gisserot-Boukhlef
user
hgissbkh
67ce9626e5cdfda52b9e883b
Duarte M. Alves
admin_assigned
2025-03-10T09:42:23.055Z
64132452d8a418df415a6ded
https://cdn-avatars.huggingface.co/v1/production/uploads/64132452d8a418df415a6ded/qkjL5G89uldHUXlCI3n4f.jpeg
Duarte Alves
user
DuarteMRAlves
67ce9626e5cdfda52b9e883c
André Martins
67ce9626e5cdfda52b9e883d
Ayoub Hammal
admin_assigned
2025-03-10T09:42:42.527Z
63937b399762cdd66be2a32f
/avatars/7aefd888a3c54673d5881dcef61f771b.svg
Ayoub Hammal
user
ayoubhammal
67ce9626e5cdfda52b9e883e
Caio Corro
admin_assigned
2025-03-10T09:42:48.603Z
677bedd522ca8585ede98470
/avatars/54bca410c446610f02aca55918c74518.svg
Caio Corro
user
caiocorro
67ce9626e5cdfda52b9e883f
Céline Hudelot
admin_assigned
2025-03-10T09:51:41.273Z
61efea03a57920a251ec19b8
/avatars/f47c8e3cb17a2bf7d43f2c152bb86885.svg
Celine Hudelot
user
CelineH
67ce9626e5cdfda52b9e8840
Emmanuel Malherbe
admin_assigned
2025-03-10T09:51:47.996Z
66f2d6a684a241caac8e16dc
/avatars/81acb87c2b07bea938251b40a2139911.svg
Emmanuel Malherbe
user
emmanuelmalherbe
67ce9626e5cdfda52b9e8841
Etienne Malaboeuf
67ce9626e5cdfda52b9e8842
Fanny Jourdan
admin_assigned
2025-03-10T09:52:09.223Z
6708db59caf70ddea8e1355d
https://cdn-avatars.huggingface.co/v1/production/uploads/6708db59caf70ddea8e1355d/C6T16AdpqoeWCk7Gg9wSH.jpeg
Fanny Jourdan
user
Fannyjrd
67ce9626e5cdfda52b9e8843
Gabriel Hautreux
admin_assigned
2025-03-10T09:52:15.512Z
67cafedda972115e89972cd7
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/P_xComqG9IttvluN-6tyB.png
Gabriel Hautreux
user
GabrielHau
67ce9626e5cdfda52b9e8844
João Alves
admin_assigned
2025-03-10T09:52:22.630Z
6772bde5c997eeb5550e80ea
/avatars/8134a4d9330317e748dc7b33e1bb25f6.svg
João Alves
user
albusonrails
67ce9626e5cdfda52b9e8845
Kevin El-Haddad
admin_assigned
2025-03-10T09:52:31.191Z
66e2c22d7cc3edd60d725267
/avatars/b217c5708c7dba8b1c220f37984ccc1e.svg
Kevin El Haddad
user
kelhad
67ce9626e5cdfda52b9e8846
Manuel Faysse
admin_assigned
2025-03-10T09:52:38.114Z
60f2e021adf471cbdf8bb660
https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
Manuel Faysse
user
manu
67ce9626e5cdfda52b9e8847
Maxime Peyrard
admin_assigned
2025-03-10T09:52:44.389Z
6369394dd322a76e1ea4bdf6
/avatars/a4e5ab0167025fbbfc970d54630ce754.svg
Maxime Peyrard
user
peyrardm
67ce9626e5cdfda52b9e8848
Nuno M. Guerreiro
admin_assigned
2025-03-10T09:52:54.367Z
67b622d2df3a86fbca306c43
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lNlshrl56oaKslArMzSzj.png
Nuno  Guerreiro
user
nunogj
67ce9626e5cdfda52b9e8849
Patrick Fernandes
67ce9626e5cdfda52b9e884a
Ricardo Rei
claimed_verified
2025-03-16T21:18:40.390Z
60d9a9791fa5d458da77754d
https://cdn-avatars.huggingface.co/v1/production/uploads/1624887555363-60d9a9791fa5d458da77754d.jpeg
Ricardo Rei
user
RicardoRei
67ce9626e5cdfda52b9e884b
Pierre Colombo
admin_assigned
2025-03-10T09:41:26.353Z
644a900e3a619fe72b14af0f
/avatars/e2d5dac3d92757ed48e37e126a3464a3.svg
Colombo
user
PierreColombo
EuroBERT: Scaling Multilingual Encoders for European Languages
General-purpose multilingual vector representations, used in retrieval,
regression and classification, are traditionally obtained from bidirectional
encoder models. Despite their wide applicability, encoders have been recently
overshadowed by advances in generative decoder-only models. However, many
innovations driving this progress are not inherently tied to decoders. In this
paper, we revisit the development of multilingual encoders through the lens of
these advances, and introduce EuroBERT, a family of multilingual encoders
covering European and widely spoken global languages. Our models outperform
existing alternatives across a diverse range of tasks, spanning multilingual
capabilities, mathematics, and coding, and natively supporting sequences of up
to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering
insights into our dataset composition and training pipeline. We publicly
release the EuroBERT models, including intermediate training checkpoints,
together with our training framework.
EuroBERT, a family of multilingual encoders covering European and global languages, outperforms existing models across various tasks and supports long sequences, surpassing traditional bidirectional encoders.
bidirectional encoder models generative decoder-only models multilingual encoders EuroBERT multilingual capabilities token sequences