Block Diffusion: Interpolating Between Autoregressive and Diffusion
  Language Models
Diffusion language models offer unique benefits over autoregressive models
due to their potential for parallelized generation and controllability, yet
they lag in likelihood modeling and are limited to fixed-length generation. In
this work, we introduce a class of block diffusion language models that
interpolate between discrete denoising diffusion and autoregressive models.
Block diffusion overcomes key limitations of both approaches by supporting
flexible-length generation and improving inference efficiency with KV caching
and parallel token sampling. We propose a recipe for building effective block
diffusion models that includes an efficient training algorithm, estimators of
gradient variance, and data-driven noise schedules to minimize the variance.
Block diffusion sets a new state-of-the-art performance among diffusion models
on language modeling benchmarks and enables generation of arbitrary-length
sequences. We provide the code, along with the model weights and blog post on
the project page: https://m-arriola.com/bd3lms/
67d2511e7d0fc37e67269f85
Marianne Arriola
admin_assigned
2025-03-14T11:16:23.456Z
660d8a40e8379a6f240193de
/avatars/64a99f756e7b5e15463c8265f3cc35c7.svg
Marianne Arriola
user
marriola
67d2511e7d0fc37e67269f86
Aaron Gokaslan
claimed_verified
2025-03-14T08:58:52.445Z
620573d0522e40b4a18d8763
/avatars/9353c064ef8ccac84d0397411d38fa90.svg
Aaron Gokaslan
user
Skylion007
67d2511e7d0fc37e67269f87
Justin T Chiu
admin_assigned
2025-03-14T11:16:30.464Z
629f763f540c0da640ae7e70
/avatars/1e8005eca079748365e7243a5d100fe9.svg
Justin T Chiu
user
justintchiu
67d2511e7d0fc37e67269f88
Zhihan Yang
admin_assigned
2025-03-14T11:16:37.189Z
6723622b2f49e07c8a4bea78
https://cdn-avatars.huggingface.co/v1/production/uploads/6723622b2f49e07c8a4bea78/w5BdcmfUrnmiykzths639.png
Zhihan Yang
user
zhihanyang
67d2511e7d0fc37e67269f89
Zhixuan Qi
admin_assigned
2025-03-14T11:16:43.642Z
6600827b4d7ed9ef947597af
/avatars/6d9c6b3ae91bfa8915c5e7bc516dcdb1.svg
Zhixuan Qi
user
Nauxihziq
67d2511e7d0fc37e67269f8a
Jiaqi Han
admin_assigned
2025-03-14T11:16:51.044Z
66ce1661d620749ad81dda84
/avatars/e8a454c3d45b2e878388b1a6e466a5a1.svg
Jiaqi Han
user
jiaqihan99
67d2511e7d0fc37e67269f8b
Subham Sekhar Sahoo
admin_assigned
2025-03-14T11:16:56.803Z
661839d73b412cdc851299c1
https://cdn-avatars.huggingface.co/v1/production/uploads/661839d73b412cdc851299c1/xicwANPQPTFdWfblisL2-.png
Subham Sekhar Sahoo
user
s-sahoo
67d2511e7d0fc37e67269f8c
Volodymyr Kuleshov
admin_assigned
2025-03-14T11:17:03.227Z
640ea1c43c82bd463ee80e19
/avatars/3213d2e8a433a207dfb96a35f0f52a92.svg
Volodymyr Kuleshov
user
kuleshov
Block Diffusion: Interpolating Between Autoregressive and Diffusion
  Language Models
Diffusion language models offer unique benefits over autoregressive models
due to their potential for parallelized generation and controllability, yet
they lag in likelihood modeling and are limited to fixed-length generation. In
this work, we introduce a class of block diffusion language models that
interpolate between discrete denoising diffusion and autoregressive models.
Block diffusion overcomes key limitations of both approaches by supporting
flexible-length generation and improving inference efficiency with KV caching
and parallel token sampling. We propose a recipe for building effective block
diffusion models that includes an efficient training algorithm, estimators of
gradient variance, and data-driven noise schedules to minimize the variance.
Block diffusion sets a new state-of-the-art performance among diffusion models
on language modeling benchmarks and enables generation of arbitrary-length
sequences. We provide the code, along with the model weights and blog post on
the project page: https://m-arriola.com/bd3lms/
Block diffusion language models improve generation efficiency and sequence length flexibility compared to autoregressive and discrete denoising diffusion models.
diffusion language models autoregressive models denoising diffusion block diffusion parallelized generation controllability likelihood modeling fixed-length generation flexible-length generation inference efficiency KV caching parallel token sampling efficient training algorithm gradient variance estimators data-driven noise schedules language modeling benchmarks arbitrary-length sequences