GPT or BERT: why not both?
We present a simple way to merge masked language modeling with causal
language modeling. This hybrid training objective results in a model that
combines the strengths of both modeling paradigms within a single transformer
stack: GPT-BERT can be transparently used like any standard causal or masked
language model. We test the pretraining process that enables this flexible
behavior on the BabyLM Challenge 2024. The results show that the hybrid
pretraining outperforms masked-only or causal-only models. We openly release
the models, training corpora and code.
6728d4bba53a076e4bafac1c
Lucas Georges Gabriel Charpentier
admin_assigned
2024-11-04T21:44:39.123Z
64c85db9b18936750d4cb690
/avatars/1f1f6df531300d6396f4d117cc764326.svg
Lucas Charpentier
user
lgcharpe
6728d4bba53a076e4bafac1d
David Samuel
admin_assigned
2024-11-05T07:53:30.031Z
63557132525beaee688cb548
/avatars/c67c0b1d9ea0f8a42c5c8b8cf8c20597.svg
David Samuel
user
davda54
GPT or BERT: why not both?
We present a simple way to merge masked language modeling with causal
language modeling. This hybrid training objective results in a model that
combines the strengths of both modeling paradigms within a single transformer
stack: GPT-BERT can be transparently used like any standard causal or masked
language model. We test the pretraining process that enables this flexible
behavior on the BabyLM Challenge 2024. The results show that the hybrid
pretraining outperforms masked-only or causal-only models. We openly release
the models, training corpora and code.
A hybrid model combining masked and causal language modeling within a transformer stack outperforms models using either paradigm alone.
masked language modeling causal language modeling transformer stack GPT-BERT BabyLM Challenge pretraining process