dInfer: An Efficient Inference Framework for Diffusion Language Models
Diffusion-based large language models (dLLMs) have emerged as a promising
alternative to autoregressive (AR) LLMs, leveraging denoising-based generation
to enable inherent parallelism. Even more and more open-sourced dLLM models
emerge, yet their widespread adoption remains constrained by the lack of a
standardized and efficient inference framework. We present dInfer, an efficient
and extensible framework for dLLM inference. dInfer decomposes the inference
pipeline into four modular components--model, diffusion iteration manager,
decoding strategy, and KV-cache manager--and integrates novel algorithms for
each component alongside system-level optimizations. Through this combination
of algorithmic innovations and system enhancements, dInfer achieves substantial
efficiency gains without compromising output quality on LLaDA-MoE. At batch
size 1, it surpasses 1,100 tokens per second on HumanEval and averages over 800
tokens per second across six benchmarks on 8times H800 GPUs. Compared to
prior systems, dInfer delivers a 10times speedup over Fast-dLLM while
maintaining similar model performance. Even compared to the AR model (with a
comparable number of activation parameters and performance) QWen2.5-3B, which
is highly optimized with the latest vLLM inference engine, dInfer still
delivers a 2-3times speedup. The implementation of dInfer is open-sourced
at https://github.com/inclusionAI/dInfer.
68efa7065178b1b97206b27d
Yuxin Ma
68efa7065178b1b97206b27e
Lun Du
68efa7065178b1b97206b27f
Lanning Wei
68efa7065178b1b97206b280
Kun Chen
68efa7065178b1b97206b281
Qian Xu
68efa7065178b1b97206b282
Kangyu Wang
68efa7065178b1b97206b283
Guofeng Feng
68efa7065178b1b97206b284
Guoshan Lu
68efa7065178b1b97206b285
Lin Liu
68efa7065178b1b97206b286
Xiaojing Qi
68efa7065178b1b97206b287
Xinyuan Zhang
68efa7065178b1b97206b288
Zhen Tao
68efa7065178b1b97206b289
Haibo Feng
68efa7065178b1b97206b28a
Ziyun Jiang
68efa7065178b1b97206b28b
Ying Xu
68efa7065178b1b97206b28c
Zenan Huang
68efa7065178b1b97206b28d
Yihong Zhuang
68efa7065178b1b97206b28e
Haokai Xu
68efa7065178b1b97206b28f
Jiaqi Hu
68efa7065178b1b97206b290
Zhenzhong Lan
68efa7065178b1b97206b291
Junbo Zhao
68efa7065178b1b97206b292
Jianguo Li
68efa7065178b1b97206b293
Da Zheng
dInfer: An Efficient Inference Framework for Diffusion Language Models
Diffusion-based large language models (dLLMs) have emerged as a promising
alternative to autoregressive (AR) LLMs, leveraging denoising-based generation
to enable inherent parallelism. Even more and more open-sourced dLLM models
emerge, yet their widespread adoption remains constrained by the lack of a
standardized and efficient inference framework. We present dInfer, an efficient
and extensible framework for dLLM inference. dInfer decomposes the inference
pipeline into four modular components--model, diffusion iteration manager,
decoding strategy, and KV-cache manager--and integrates novel algorithms for
each component alongside system-level optimizations. Through this combination
of algorithmic innovations and system enhancements, dInfer achieves substantial
efficiency gains without compromising output quality on LLaDA-MoE. At batch
size 1, it surpasses 1,100 tokens per second on HumanEval and averages over 800
tokens per second across six benchmarks on 8times H800 GPUs. Compared to
prior systems, dInfer delivers a 10times speedup over Fast-dLLM while
maintaining similar model performance. Even compared to the AR model (with a
comparable number of activation parameters and performance) QWen2.5-3B, which
is highly optimized with the latest vLLM inference engine, dInfer still
delivers a 2-3times speedup. The implementation of dInfer is open-sourced
at https://github.com/inclusionAI/dInfer.
dInfer is an efficient and extensible framework for diffusion-based large language model inference, achieving significant speedups over existing systems without compromising output quality.
diffusion-based large language models autoregressive LLMs denoising-based generation inference framework modular components diffusion iteration manager decoding strategy KV-cache manager algorithmic innovations system-level optimizations LLaDA-MoE HumanEval H800 GPUs Fast-dLLM QWen2.5-3B vLLM inference engine