Large Language Diffusion Models
Autoregressive models (ARMs) are widely regarded as the cornerstone of large
language models (LLMs). We challenge this notion by introducing LLaDA, a
diffusion model trained from scratch under the pre-training and supervised
fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data
masking process and a reverse process, parameterized by a vanilla Transformer
to predict masked tokens. By optimizing a likelihood bound, it provides a
principled generative approach for probabilistic inference. Across extensive
benchmarks, LLaDA demonstrates strong scalability, outperforming our
self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong
LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive
instruction-following abilities in case studies such as multi-turn dialogue.
Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal
poem completion task. Our findings establish diffusion models as a viable and
promising alternative to ARMs, challenging the assumption that key LLM
capabilities discussed above are inherently tied to ARMs.
67b2c31125f77e5fc242f4f8
Shen Nie
admin_assigned
2025-02-28T12:48:04.969Z
640dd9a5fdeaae13908208a7
/avatars/61f8f1d5f6ef1c4af1f47285e9cc0217.svg
nieshen
user
nieshen
67b2c31125f77e5fc242f4f9
Fengqi Zhu
67b2c31125f77e5fc242f4fa
Zebin You
admin_assigned
2025-02-28T12:47:21.498Z
624f909eac5dd186b01ac3f5
/avatars/0aafdb1cbb492fda52a0303031cc6c14.svg
Zebin You
user
yyyou
67b2c31125f77e5fc242f4fb
Xiaolu Zhang
admin_assigned
2025-02-28T12:47:27.956Z
67513d6d3b8586521cda5d76
/avatars/0f95cc5c23a0a1da289aa785bd33b616.svg
Xiaolu  Zhang
user
xiaolu0714
67b2c31125f77e5fc242f4fc
Jingyang Ou
admin_assigned
2025-02-19T15:51:46.024Z
656949b71d7c2ca7b7aae5f2
/avatars/e7b23e260eb348cc26b849aaa601a503.svg
Jingyang Ou
user
JingyangOu
67b2c31125f77e5fc242f4fd
Jun Hu
67b2c31125f77e5fc242f4fe
Jun Zhou
67b2c31125f77e5fc242f4ff
Yankai Lin
admin_assigned
2025-02-19T15:51:39.766Z
657a651e1433ea7d44de6397
/avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
Yankai Lin
user
lyk423
67b2c31125f77e5fc242f500
Ji-Rong Wen
admin_assigned
2025-02-19T15:51:33.428Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
67b2c31125f77e5fc242f501
Chongxuan Li
admin_assigned
2025-02-19T15:51:26.668Z
64c07b488e2612254361153b
/avatars/ade0f783cc4c2d3e73f402637f595471.svg
chongxuan li
user
zhenxuan00
Large Language Diffusion Models
Autoregressive models (ARMs) are widely regarded as the cornerstone of large
language models (LLMs). We challenge this notion by introducing LLaDA, a
diffusion model trained from scratch under the pre-training and supervised
fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data
masking process and a reverse process, parameterized by a vanilla Transformer
to predict masked tokens. By optimizing a likelihood bound, it provides a
principled generative approach for probabilistic inference. Across extensive
benchmarks, LLaDA demonstrates strong scalability, outperforming our
self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong
LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive
instruction-following abilities in case studies such as multi-turn dialogue.
Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal
poem completion task. Our findings establish diffusion models as a viable and
promising alternative to ARMs, challenging the assumption that key LLM
capabilities discussed above are inherently tied to ARMs.
LLaDA, a diffusion model trained from scratch, outperforms autoregressive models in benchmarks and demonstrates strong instruction-following capabilities, challenging the dominance of ARMs in LLMs.
autoregressive models LLaDA diffusion model pre-training supervised fine-tuning vanilla Transformer likelihood bound probabilistic inference in-context learning instruction-following reversal curse LLaMA3 GPT-4o reversal poem completion