Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion
  Forcing
Diffusion Large Language Models (dLLMs) have emerged as a promising
alternative to autoregressive (AR) LLMs for text generation, with the potential
to decode multiple tokens in a single iteration. However, none of the existing
open-source dLLMs have achieved superior inference speed over AR LLMs of
similar size. This paper breaks this barrier based on a simple and effective
strategy named discrete diffusion forcing (D2F). D2F equips dLLMs with two key
capabilities: (1) block-wise autoregressive generation to enable KV cache
utilization; (2) prediction of following tokens without requiring completion of
prior blocks for inter-block parallel decoding. In this way, the vanilla dLLMs
are refurbished into an AR-diffusion hybrid paradigm for efficient inference.
D2F can be implemented with an asymmetric distillation process based on
pre-trained dLLMs. We further propose a pipelined parallel decoding algorithm,
which enables a trade-off between efficiency and efficacy. Empirically, D2F
dLLMs achieve more than 2.5times inference speed than LLaMA3 and
Qwen2.5 on GSM8K. Compared to vanilla dLLMs like LLaDA and Dream, the
acceleration can be more than 50times while maintaining comparable
output quality. The code is available at
https://github.com/zhijie-group/Discrete-Diffusion-Forcing.
689d3293b083e610d741e992
Xu Wang
689d3293b083e610d741e993
Chenkai Xu
689d3293b083e610d741e994
Yijie Jin
claimed_verified
2025-08-14T13:37:50.550Z
66d05337e62d6bbf50186c2f
/avatars/f5be15e754f0fbbb37d2cc5ea417f729.svg
Yijie Jin
user
DrewJin0827
689d3293b083e610d741e995
Jiachun Jin
689d3293b083e610d741e996
Hao Zhang
689d3293b083e610d741e997
Zhijie Deng
Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion
  Forcing
Diffusion Large Language Models (dLLMs) have emerged as a promising
alternative to autoregressive (AR) LLMs for text generation, with the potential
to decode multiple tokens in a single iteration. However, none of the existing
open-source dLLMs have achieved superior inference speed over AR LLMs of
similar size. This paper breaks this barrier based on a simple and effective
strategy named discrete diffusion forcing (D2F). D2F equips dLLMs with two key
capabilities: (1) block-wise autoregressive generation to enable KV cache
utilization; (2) prediction of following tokens without requiring completion of
prior blocks for inter-block parallel decoding. In this way, the vanilla dLLMs
are refurbished into an AR-diffusion hybrid paradigm for efficient inference.
D2F can be implemented with an asymmetric distillation process based on
pre-trained dLLMs. We further propose a pipelined parallel decoding algorithm,
which enables a trade-off between efficiency and efficacy. Empirically, D2F
dLLMs achieve more than 2.5times inference speed than LLaMA3 and
Qwen2.5 on GSM8K. Compared to vanilla dLLMs like LLaDA and Dream, the
acceleration can be more than 50times while maintaining comparable
output quality. The code is available at
https://github.com/zhijie-group/Discrete-Diffusion-Forcing.
Discrete diffusion forcing enhances diffusion large language models with block-wise autoregressive generation and inter-block parallel decoding, significantly improving inference speed while maintaining quality.
diffusion large language models autoregressive LLMs discrete diffusion forcing block-wise autoregressive generation KV cache inter-block parallel decoding asymmetric distillation pipelined parallel decoding GSM8K LLaMA3 Qwen2.5 LLaDA Dream