A Survey on Diffusion Language Models
Diffusion Language Models (DLMs) are rapidly emerging as a powerful and
promising alternative to the dominant autoregressive (AR) paradigm. By
generating tokens in parallel through an iterative denoising process, DLMs
possess inherent advantages in reducing inference latency and capturing
bidirectional context, thereby enabling fine-grained control over the
generation process. While achieving a several-fold speed-up, recent
advancements have allowed DLMs to show performance comparable to their
autoregressive counterparts, making them a compelling choice for various
natural language processing tasks. In this survey, we provide a holistic
overview of the current DLM landscape. We trace its evolution and relationship
with other paradigms, such as autoregressive and masked language models, and
cover both foundational principles and state-of-the-art models. Our work offers
an up-to-date, comprehensive taxonomy and an in-depth analysis of current
techniques, from pre-training strategies to advanced post-training methods.
Another contribution of this survey is a thorough review of DLM inference
strategies and optimizations, including improvements in decoding parallelism,
caching mechanisms, and generation quality. We also highlight the latest
approaches to multimodal extensions of DLMs and delineate their applications
across various practical scenarios. Furthermore, our discussion addresses the
limitations and challenges of DLMs, including efficiency, long-sequence
handling, and infrastructure requirements, while outlining future research
directions to sustain progress in this rapidly evolving field. Project GitHub
is available at https://github.com/VILA-Lab/Awesome-DLMs.
689eab50a4caabb4320e5dba
Tianyi Li
689eab50a4caabb4320e5dbb
Mingda Chen
689eab50a4caabb4320e5dbc
Bowei Guo
689eab50a4caabb4320e5dbd
Zhiqiang Shen
A Survey on Diffusion Language Models
Diffusion Language Models (DLMs) are rapidly emerging as a powerful and
promising alternative to the dominant autoregressive (AR) paradigm. By
generating tokens in parallel through an iterative denoising process, DLMs
possess inherent advantages in reducing inference latency and capturing
bidirectional context, thereby enabling fine-grained control over the
generation process. While achieving a several-fold speed-up, recent
advancements have allowed DLMs to show performance comparable to their
autoregressive counterparts, making them a compelling choice for various
natural language processing tasks. In this survey, we provide a holistic
overview of the current DLM landscape. We trace its evolution and relationship
with other paradigms, such as autoregressive and masked language models, and
cover both foundational principles and state-of-the-art models. Our work offers
an up-to-date, comprehensive taxonomy and an in-depth analysis of current
techniques, from pre-training strategies to advanced post-training methods.
Another contribution of this survey is a thorough review of DLM inference
strategies and optimizations, including improvements in decoding parallelism,
caching mechanisms, and generation quality. We also highlight the latest
approaches to multimodal extensions of DLMs and delineate their applications
across various practical scenarios. Furthermore, our discussion addresses the
limitations and challenges of DLMs, including efficiency, long-sequence
handling, and infrastructure requirements, while outlining future research
directions to sustain progress in this rapidly evolving field. Project GitHub
is available at https://github.com/VILA-Lab/Awesome-DLMs.
Diffusion Language Models offer parallel token generation, reducing inference latency and capturing bidirectional context, and are compared to autoregressive models in various NLP tasks.
Diffusion Language Models autoregressive denoising process inference latency bidirectional context pre-training strategies post-training methods decoding parallelism caching mechanisms generation quality multimodal extensions