Quantization Meets dLLMs: A Systematic Study of Post-training
  Quantization for Diffusion LLMs
Recent advances in diffusion large language models (dLLMs) have introduced a
promising alternative to autoregressive (AR) LLMs for natural language
generation tasks, leveraging full attention and denoising-based decoding
strategies. However, the deployment of these models on edge devices remains
challenging due to their massive parameter scale and high resource demands.
While post-training quantization (PTQ) has emerged as a widely adopted
technique for compressing AR LLMs, its applicability to dLLMs remains largely
unexplored. In this work, we present the first systematic study on quantizing
diffusion-based language models. We begin by identifying the presence of
activation outliers, characterized by abnormally large activation values that
dominate the dynamic range. These outliers pose a key challenge to low-bit
quantization, as they make it difficult to preserve precision for the majority
of values. More importantly, we implement state-of-the-art PTQ methods and
conduct a comprehensive evaluation across multiple task types and model
variants. Our analysis is structured along four key dimensions: bit-width,
quantization method, task category, and model type. Through this
multi-perspective evaluation, we offer practical insights into the quantization
behavior of dLLMs under different configurations. We hope our findings provide
a foundation for future research in efficient dLLM deployment. All codes and
experimental setups will be released to support the community.
68a69cf99e4b49496aac6a7f
Haokun Lin
admin_assigned
2025-08-21T10:40:58.332Z
6841378dcb9938c7aed795be
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/WNI8gQVHdWyt0OvmRINDp.png
Haokun Lin
user
Felix1023
68a69cf99e4b49496aac6a80
Haobo Xu
68a69cf99e4b49496aac6a81
Yichen Wu
admin_assigned
2025-08-21T10:41:33.065Z
67b05af6626cd81034211436
/avatars/c2f13727955fbb2b8f1df2df3972e417.svg
Yichen Wu
user
chriswyc
68a69cf99e4b49496aac6a82
Ziyu Guo
admin_assigned
2025-08-21T10:41:38.881Z
647d9ab61a1fcad2fdbf2d3d
/avatars/48c8aeae8979d2c87df8bde922437d62.svg
Ziyu Guo
user
ZiyuG
68a69cf99e4b49496aac6a83
Renrui Zhang
68a69cf99e4b49496aac6a84
Zhichao Lu
68a69cf99e4b49496aac6a85
Ying Wei
68a69cf99e4b49496aac6a86
Qingfu Zhang
admin_assigned
2025-08-21T10:42:17.485Z
64478bf230fa4ecb85ddf661
/avatars/16ffa30a8785b20c8fdb61021d385070.svg
Qingfu Zhang
user
cityug7353
68a69cf99e4b49496aac6a87
Zhenan Sun
Quantization Meets dLLMs: A Systematic Study of Post-training
  Quantization for Diffusion LLMs
Recent advances in diffusion large language models (dLLMs) have introduced a
promising alternative to autoregressive (AR) LLMs for natural language
generation tasks, leveraging full attention and denoising-based decoding
strategies. However, the deployment of these models on edge devices remains
challenging due to their massive parameter scale and high resource demands.
While post-training quantization (PTQ) has emerged as a widely adopted
technique for compressing AR LLMs, its applicability to dLLMs remains largely
unexplored. In this work, we present the first systematic study on quantizing
diffusion-based language models. We begin by identifying the presence of
activation outliers, characterized by abnormally large activation values that
dominate the dynamic range. These outliers pose a key challenge to low-bit
quantization, as they make it difficult to preserve precision for the majority
of values. More importantly, we implement state-of-the-art PTQ methods and
conduct a comprehensive evaluation across multiple task types and model
variants. Our analysis is structured along four key dimensions: bit-width,
quantization method, task category, and model type. Through this
multi-perspective evaluation, we offer practical insights into the quantization
behavior of dLLMs under different configurations. We hope our findings provide
a foundation for future research in efficient dLLM deployment. All codes and
experimental setups will be released to support the community.
A systematic study on quantizing diffusion large language models identifies challenges and evaluates state-of-the-art methods across various configurations to improve deployment on edge devices.
diffusion large language models autoregressive LLMs full attention denoising-based decoding post-training quantization activation outliers low-bit quantization bit-width quantization method task category model type