ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on
  Transformer Encoder Models Performance
Pretrained transformer-encoder models like DeBERTaV3 and ModernBERT introduce
architectural advancements aimed at improving efficiency and performance.
Although the authors of ModernBERT report improved performance over DeBERTaV3
on several benchmarks, the lack of disclosed training data and the absence of
comparisons using a shared dataset make it difficult to determine whether these
gains are due to architectural improvements or differences in training data. In
this work, we conduct a controlled study by pretraining ModernBERT on the same
dataset as CamemBERTaV2, a DeBERTaV3 French model, isolating the effect of
model design. Our results show that the previous model generation remains
superior in sample efficiency and overall benchmark performance, with
ModernBERT's primary advantage being faster training and inference speed.
However, the new proposed model still provides meaningful architectural
improvements compared to earlier models such as BERT and RoBERTa. Additionally,
we observe that high-quality pre-training data accelerates convergence but does
not significantly improve final performance, suggesting potential benchmark
saturation. These findings show the importance of disentangling pretraining
data from architectural innovations when evaluating transformer models.
67fca0ca05cd5b5035123b7e
Wissam Antoun
claimed_verified
2025-04-14T18:40:06.158Z
605350699008a64d7b43b78d
https://cdn-avatars.huggingface.co/v1/production/uploads/1616431735085-605350699008a64d7b43b78d.jpeg
Wissam Antoun
user
wissamantoun
67fca0ca05cd5b5035123b7f
Benoît Sagot
67fca0ca05cd5b5035123b80
Djamé Seddah
ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on
  Transformer Encoder Models Performance
Pretrained transformer-encoder models like DeBERTaV3 and ModernBERT introduce
architectural advancements aimed at improving efficiency and performance.
Although the authors of ModernBERT report improved performance over DeBERTaV3
on several benchmarks, the lack of disclosed training data and the absence of
comparisons using a shared dataset make it difficult to determine whether these
gains are due to architectural improvements or differences in training data. In
this work, we conduct a controlled study by pretraining ModernBERT on the same
dataset as CamemBERTaV2, a DeBERTaV3 French model, isolating the effect of
model design. Our results show that the previous model generation remains
superior in sample efficiency and overall benchmark performance, with
ModernBERT's primary advantage being faster training and inference speed.
However, the new proposed model still provides meaningful architectural
improvements compared to earlier models such as BERT and RoBERTa. Additionally,
we observe that high-quality pre-training data accelerates convergence but does
not significantly improve final performance, suggesting potential benchmark
saturation. These findings show the importance of disentangling pretraining
data from architectural innovations when evaluating transformer models.
A controlled study comparing ModernBERT with CamemBERTaV2 demonstrates that the original model design remains superior in sample efficiency and benchmark performance, though ModernBERT offers faster training and inference.
transformer-encoder models DeBERTaV3 ModernBERT CamemBERTaV2 sample efficiency benchmark performance pretraining data architectural innovations transformer models BERT RoBERTa high-quality pre-training data convergence benchmark saturation