Simple and Effective Masked Diffusion Language Models
While diffusion models excel at generating high-quality images, prior work
reports a significant performance gap between diffusion and autoregressive (AR)
methods in language modeling. In this work, we show that simple masked discrete
diffusion is more performant than previously thought. We apply an effective
training recipe that improves the performance of masked diffusion models and
derive a simplified, Rao-Blackwellized objective that results in additional
improvements. Our objective has a simple form -- it is a mixture of classical
masked language modeling losses -- and can be used to train encoder-only
language models that admit efficient samplers, including ones that can generate
arbitrary lengths of text semi-autoregressively like a traditional language
model. On language modeling benchmarks, a range of masked diffusion models
trained with modern engineering practices achieves a new state-of-the-art among
diffusion models, and approaches AR perplexity. We release our code at:
https://github.com/kuleshov-group/mdlm
66690f02aa74814619caee13
Subham Sekhar Sahoo
admin_assigned
2024-06-12T08:36:17.388Z
661839d73b412cdc851299c1
https://cdn-avatars.huggingface.co/v1/production/uploads/661839d73b412cdc851299c1/xicwANPQPTFdWfblisL2-.png
Subham Sekhar Sahoo
user
s-sahoo
66690f02aa74814619caee14
Marianne Arriola
admin_assigned
2024-06-12T08:36:23.171Z
660d8a40e8379a6f240193de
/avatars/64a99f756e7b5e15463c8265f3cc35c7.svg
Marianne Arriola
user
marriola
66690f02aa74814619caee15
Yair Schiff
admin_assigned
2024-06-12T08:36:28.591Z
6418e5e722270b3ccf15dfef
https://cdn-avatars.huggingface.co/v1/production/uploads/6418e5e722270b3ccf15dfef/ZiammTpL9FVMuE7DN8n5B.jpeg
Yair Schiff
user
yairschiff
66690f02aa74814619caee16
Aaron Gokaslan
claimed_verified
2024-06-12T07:29:31.761Z
620573d0522e40b4a18d8763
/avatars/9353c064ef8ccac84d0397411d38fa90.svg
Aaron Gokaslan
user
Skylion007
66690f02aa74814619caee17
Edgar Marroquin
admin_assigned
2024-06-12T08:36:41.104Z
6616e84eceda8584fd0c109f
/avatars/5261c6223571955260f7720ff88a8d67.svg
Edgar Marroquin
user
emarro
66690f02aa74814619caee18
Justin T Chiu
admin_assigned
2024-06-12T08:36:46.781Z
629f763f540c0da640ae7e70
/avatars/1e8005eca079748365e7243a5d100fe9.svg
Justin T Chiu
user
justintchiu
66690f02aa74814619caee19
Alexander Rush
admin_assigned
2024-06-12T08:36:58.268Z
6372605b785dc489da5a037a
https://cdn-avatars.huggingface.co/v1/production/uploads/1668440123408-noauth.png
Alex Rush
user
OfficialAL33900
66690f02aa74814619caee1a
Volodymyr Kuleshov
admin_assigned
2024-06-12T08:37:04.642Z
640ea1c43c82bd463ee80e19
/avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
Volodymyr Kuleshov
user
kuleshov
Simple and Effective Masked Diffusion Language Models
While diffusion models excel at generating high-quality images, prior work
reports a significant performance gap between diffusion and autoregressive (AR)
methods in language modeling. In this work, we show that simple masked discrete
diffusion is more performant than previously thought. We apply an effective
training recipe that improves the performance of masked diffusion models and
derive a simplified, Rao-Blackwellized objective that results in additional
improvements. Our objective has a simple form -- it is a mixture of classical
masked language modeling losses -- and can be used to train encoder-only
language models that admit efficient samplers, including ones that can generate
arbitrary lengths of text semi-autoregressively like a traditional language
model. On language modeling benchmarks, a range of masked diffusion models
trained with modern engineering practices achieves a new state-of-the-art among
diffusion models, and approaches AR perplexity. We release our code at:
https://github.com/kuleshov-group/mdlm
Masked discrete diffusion models achieve improved performance on language modeling benchmarks through a simplified training recipe and objective, approaching autoregressive models in perplexity.
diffusion models masked diffusion autoregressive methods masked discrete diffusion Rao-Blackwellized objective encoder-only language models semi-autoregressive generation language modeling benchmarks perplexity