Discrete Diffusion in Large Language and Multimodal Models: A Survey
In this work, we provide a systematic survey of Discrete Diffusion Language
Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs).
Unlike autoregressive (AR) models, dLLMs and dMLLMs adopt a multi-token,
parallel decoding paradigm using full attention and a denoising-based
generation strategy. This paradigm naturally enables parallel generation,
fine-grained output controllability, and dynamic, response-aware perception.
These capabilities are previously difficult to achieve with AR models.
Recently, a growing number of industrial-scale proprietary d(M)LLMs, as well as
a large number of open-source academic d(M)LLMs, have demonstrated performance
comparable to their autoregressive counterparts, while achieving up to 10x
acceleration in inference speed.
  The advancement of discrete diffusion LLMs and MLLMs has been largely driven
by progress in two domains. The first is the development of autoregressive LLMs
and MLLMs, which has accumulated vast amounts of data, benchmarks, and
foundational infrastructure for training and inference. The second contributing
domain is the evolution of the mathematical models underlying discrete
diffusion. Together, these advancements have catalyzed a surge in dLLMs and
dMLLMs research in early 2025.
  In this work, we present a comprehensive overview of the research in the dLLM
and dMLLM domains. We trace the historical development of dLLMs and dMLLMs,
formalize the underlying mathematical frameworks, and categorize representative
models. We further analyze key techniques for training and inference, and
summarize emerging applications across language, vision-language, and
biological domains. We conclude by discussing future directions for research
and deployment.
  Paper collection: https://github.com/LiQiiiii/DLLM-Survey
6850ccab5e07650ecce88fd7
Runpeng Yu
6850ccab5e07650ecce88fd8
Qi Li
6850ccab5e07650ecce88fd9
Xinchao Wang
Discrete Diffusion in Large Language and Multimodal Models: A Survey
In this work, we provide a systematic survey of Discrete Diffusion Language
Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs).
Unlike autoregressive (AR) models, dLLMs and dMLLMs adopt a multi-token,
parallel decoding paradigm using full attention and a denoising-based
generation strategy. This paradigm naturally enables parallel generation,
fine-grained output controllability, and dynamic, response-aware perception.
These capabilities are previously difficult to achieve with AR models.
Recently, a growing number of industrial-scale proprietary d(M)LLMs, as well as
a large number of open-source academic d(M)LLMs, have demonstrated performance
comparable to their autoregressive counterparts, while achieving up to 10x
acceleration in inference speed.
  The advancement of discrete diffusion LLMs and MLLMs has been largely driven
by progress in two domains. The first is the development of autoregressive LLMs
and MLLMs, which has accumulated vast amounts of data, benchmarks, and
foundational infrastructure for training and inference. The second contributing
domain is the evolution of the mathematical models underlying discrete
diffusion. Together, these advancements have catalyzed a surge in dLLMs and
dMLLMs research in early 2025.
  In this work, we present a comprehensive overview of the research in the dLLM
and dMLLM domains. We trace the historical development of dLLMs and dMLLMs,
formalize the underlying mathematical frameworks, and categorize representative
models. We further analyze key techniques for training and inference, and
summarize emerging applications across language, vision-language, and
biological domains. We conclude by discussing future directions for research
and deployment.
  Paper collection: https://github.com/LiQiiiii/DLLM-Survey
Discrete Diffusion Language Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs) enable parallel generation and faster inference compared to autoregressive models through denoising-based strategies and full attention mechanisms.
Discrete Diffusion Language Models Discrete Diffusion Multimodal Language Models autoregressive models multi-token parallel decoding full attention denoising-based generation response-aware perception inference speed autoregressive LLMs autoregressive MLLMs mathematical models historical development training inference language applications vision-language applications biological applications future research directions deployment