Should We Still Pretrain Encoders with Masked Language Modeling?
Learning high-quality text representations is fundamental to a wide range of
NLP tasks. While encoder pretraining has traditionally relied on Masked
Language Modeling (MLM), recent evidence suggests that decoder models
pretrained with Causal Language Modeling (CLM) can be effectively repurposed as
encoders, often surpassing traditional encoders on text representation
benchmarks. However, it remains unclear whether these gains reflect an inherent
advantage of the CLM objective or arise from confounding factors such as model
and data scale. In this paper, we address this question through a series of
large-scale, carefully controlled pretraining ablations, training a total of 30
models ranging from 210 million to 1 billion parameters, and conducting over
15,000 fine-tuning and evaluation runs. We find that while training with MLM
generally yields better performance across text representation tasks,
CLM-trained models are more data-efficient and demonstrate improved fine-tuning
stability. Building on these findings, we experimentally show that a biphasic
training strategy that sequentially applies CLM and then MLM, achieves optimal
performance under a fixed computational training budget. Moreover, we
demonstrate that this strategy becomes more appealing when initializing from
readily available pretrained CLM models (from the existing LLM ecosystem),
reducing the computational burden needed to train best-in-class encoder models.
We release all project artifacts at https://hf.co/MLMvsCLM to foster further
research.
6864e267d59a9eda59024bab
Hippolyte Gisserot-Boukhlef
claimed_verified
2025-07-05T07:53:51.109Z
65fa95405355a52c784633fc
https://cdn-avatars.huggingface.co/v1/production/uploads/65fa95405355a52c784633fc/rSfBUHPa7eSAsLd8DuOq4.png
Hippolyte Gisserot-Boukhlef
user
hgissbkh
6864e267d59a9eda59024bac
Nicolas Boizard
claimed_verified
2025-07-02T12:22:12.240Z
62be186a5f59ff2320e6e32b
https://cdn-avatars.huggingface.co/v1/production/uploads/62be186a5f59ff2320e6e32b/W_emoC2uItM-MJZyCfIKI.png
Nicolas-BZRD
user
Nicolas-BZRD
6864e267d59a9eda59024bad
Manuel Faysse
admin_assigned
2025-07-08T13:52:55.139Z
60f2e021adf471cbdf8bb660
https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
Manuel Faysse
user
manu
6864e267d59a9eda59024bae
Duarte M. Alves
6864e267d59a9eda59024baf
Emmanuel Malherbe
admin_assigned
2025-07-08T13:53:10.878Z
66f2d6a684a241caac8e16dc
/avatars/81acb87c2b07bea938251b40a2139911.svg
Emmanuel Malherbe
user
emmanuelmalherbe
6864e267d59a9eda59024bb0
André F. T. Martins
6864e267d59a9eda59024bb1
Céline Hudelot
admin_assigned
2025-07-08T13:53:24.532Z
61efea03a57920a251ec19b8
/avatars/f47c8e3cb17a2bf7d43f2c152bb86885.svg
Celine Hudelot
user
CelineH
6864e267d59a9eda59024bb2
Pierre Colombo
admin_assigned
2025-07-08T13:53:30.831Z
644a900e3a619fe72b14af0f
/avatars/e2d5dac3d92757ed48e37e126a3464a3.svg
Colombo
user
PierreColombo
Should We Still Pretrain Encoders with Masked Language Modeling?
Learning high-quality text representations is fundamental to a wide range of
NLP tasks. While encoder pretraining has traditionally relied on Masked
Language Modeling (MLM), recent evidence suggests that decoder models
pretrained with Causal Language Modeling (CLM) can be effectively repurposed as
encoders, often surpassing traditional encoders on text representation
benchmarks. However, it remains unclear whether these gains reflect an inherent
advantage of the CLM objective or arise from confounding factors such as model
and data scale. In this paper, we address this question through a series of
large-scale, carefully controlled pretraining ablations, training a total of 30
models ranging from 210 million to 1 billion parameters, and conducting over
15,000 fine-tuning and evaluation runs. We find that while training with MLM
generally yields better performance across text representation tasks,
CLM-trained models are more data-efficient and demonstrate improved fine-tuning
stability. Building on these findings, we experimentally show that a biphasic
training strategy that sequentially applies CLM and then MLM, achieves optimal
performance under a fixed computational training budget. Moreover, we
demonstrate that this strategy becomes more appealing when initializing from
readily available pretrained CLM models (from the existing LLM ecosystem),
reducing the computational burden needed to train best-in-class encoder models.
We release all project artifacts at https://hf.co/MLMvsCLM to foster further
research.
A biphasic training strategy combining Causal Language Modeling and Masked Language Modeling yields optimal text representation performance, especially when initialized with pretrained CLM models.
Masked Language Modeling Causal Language Modeling text representation encoder pretraining decoder models pretraining ablations fine-tuning stability biphasic training strategy LLM ecosystem