The Diffusion Duality
Uniform-state discrete diffusion models hold the promise of fast text
generation due to their inherent ability to self-correct. However, they are
typically outperformed by autoregressive models and masked diffusion models. In
this work, we narrow this performance gap by leveraging a key insight:
Uniform-state diffusion processes naturally emerge from an underlying Gaussian
diffusion. Our method, Duo, transfers powerful techniques from Gaussian
diffusion to improve both training and sampling. First, we introduce a
curriculum learning strategy guided by the Gaussian process, doubling training
speed by reducing variance. Models trained with curriculum learning surpass
autoregressive models in zero-shot perplexity on 3 of 7 benchmarks. Second, we
present Discrete Consistency Distillation, which adapts consistency
distillation from the continuous to the discrete setting. This algorithm
unlocks few-step generation in diffusion language models by accelerating
sampling by two orders of magnitude. We provide the code and model checkpoints
on the project page: http://s-sahoo.github.io/duo
684fb2f060b4a34dbe007aeb
Subham Sekhar Sahoo
684fb2f060b4a34dbe007aec
Justin Deschenaux
admin_assigned
2025-06-17T07:42:18.956Z
62d073f2485856cd710ed9fd
https://cdn-avatars.huggingface.co/v1/production/uploads/62d073f2485856cd710ed9fd/B6dck3fi5X9hhN4ALulMH.jpeg
Justin Deschenaux
user
jdeschena
684fb2f060b4a34dbe007aed
Aaron Gokaslan
admin_assigned
2025-06-17T07:42:24.986Z
620573d0522e40b4a18d8763
/avatars/9353c064ef8ccac84d0397411d38fa90.svg
Aaron Gokaslan
user
Skylion007
684fb2f060b4a34dbe007aee
Guanghan Wang
684fb2f060b4a34dbe007aef
Justin Chiu
684fb2f060b4a34dbe007af0
Volodymyr Kuleshov
admin_assigned
2025-06-17T07:42:38.474Z
640ea1c43c82bd463ee80e19
/avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
Volodymyr Kuleshov
user
kuleshov
The Diffusion Duality
Uniform-state discrete diffusion models hold the promise of fast text
generation due to their inherent ability to self-correct. However, they are
typically outperformed by autoregressive models and masked diffusion models. In
this work, we narrow this performance gap by leveraging a key insight:
Uniform-state diffusion processes naturally emerge from an underlying Gaussian
diffusion. Our method, Duo, transfers powerful techniques from Gaussian
diffusion to improve both training and sampling. First, we introduce a
curriculum learning strategy guided by the Gaussian process, doubling training
speed by reducing variance. Models trained with curriculum learning surpass
autoregressive models in zero-shot perplexity on 3 of 7 benchmarks. Second, we
present Discrete Consistency Distillation, which adapts consistency
distillation from the continuous to the discrete setting. This algorithm
unlocks few-step generation in diffusion language models by accelerating
sampling by two orders of magnitude. We provide the code and model checkpoints
on the project page: http://s-sahoo.github.io/duo
Duo improves uniform-state discrete diffusion models by transferring techniques from Gaussian diffusion, enhancing training speed and enabling fast few-step text generation.
discrete diffusion models Gaussian diffusion curriculum learning Discrete Consistency Distillation zero-shot perplexity few-step generation