ParallelBench: Understanding the Trade-offs of Parallel Decoding in
  Diffusion LLMs
While most autoregressive LLMs are constrained to one-by-one decoding,
diffusion LLMs (dLLMs) have attracted growing interest for their potential to
dramatically accelerate inference through parallel decoding. Despite this
promise, the conditional independence assumption in dLLMs causes parallel
decoding to ignore token dependencies, inevitably degrading generation quality
when these dependencies are strong. However, existing works largely overlook
these inherent challenges, and evaluations on standard benchmarks (e.g., math
and coding) are not sufficient to capture the quality degradation caused by
parallel decoding. To address this gap, we first provide an
information-theoretic analysis of parallel decoding. We then conduct case
studies on analytically tractable synthetic list operations from both data
distribution and decoding strategy perspectives, offering quantitative insights
that highlight the fundamental limitations of parallel decoding. Building on
these insights, we propose ParallelBench, the first benchmark specifically
designed for dLLMs, featuring realistic tasks that are trivial for humans and
autoregressive LLMs yet exceptionally challenging for dLLMs under parallel
decoding. Using ParallelBench, we systematically analyze both dLLMs and
autoregressive LLMs, revealing that: (i) dLLMs under parallel decoding can
suffer dramatic quality degradation in real-world scenarios, and (ii) current
parallel decoding strategies struggle to adapt their degree of parallelism
based on task difficulty, thus failing to achieve meaningful speedup without
compromising quality. Our findings underscore the pressing need for innovative
decoding methods that can overcome the current speed-quality trade-off. We
release our benchmark to help accelerate the development of truly efficient
dLLMs.
68f056d836f8b025381e194a
Wonjun Kang
68f056d836f8b025381e194b
Kevin Galim
68f056d836f8b025381e194c
Seunghyuk Oh
68f056d836f8b025381e194d
Minjae Lee
claimed_verified
2025-10-16T06:38:41.418Z
61ad9c3300d01045fca0ad64
/avatars/04c53c2d68d80db1053e5ebadbda5592.svg
Min Jae Lee
user
mjbooo
68f056d836f8b025381e194e
Yuchen Zeng
68f056d836f8b025381e194f
Shuibai Zhang
68f056d836f8b025381e1950
Coleman Hooper
68f056d836f8b025381e1951
Yuezhou Hu
claimed_verified
2025-10-23T15:02:14.690Z
67dc66fe55c24fc4f981a4ab
/avatars/7bd900ade802d99db7c562ad6c2f6661.svg
Yuezhou Hu
user
yuezhouhu
68f056d836f8b025381e1952
Hyung Il Koo
68f056d836f8b025381e1953
Nam Ik Cho
68f056d836f8b025381e1954
Kangwook Lee
ParallelBench: Understanding the Trade-offs of Parallel Decoding in
  Diffusion LLMs
While most autoregressive LLMs are constrained to one-by-one decoding,
diffusion LLMs (dLLMs) have attracted growing interest for their potential to
dramatically accelerate inference through parallel decoding. Despite this
promise, the conditional independence assumption in dLLMs causes parallel
decoding to ignore token dependencies, inevitably degrading generation quality
when these dependencies are strong. However, existing works largely overlook
these inherent challenges, and evaluations on standard benchmarks (e.g., math
and coding) are not sufficient to capture the quality degradation caused by
parallel decoding. To address this gap, we first provide an
information-theoretic analysis of parallel decoding. We then conduct case
studies on analytically tractable synthetic list operations from both data
distribution and decoding strategy perspectives, offering quantitative insights
that highlight the fundamental limitations of parallel decoding. Building on
these insights, we propose ParallelBench, the first benchmark specifically
designed for dLLMs, featuring realistic tasks that are trivial for humans and
autoregressive LLMs yet exceptionally challenging for dLLMs under parallel
decoding. Using ParallelBench, we systematically analyze both dLLMs and
autoregressive LLMs, revealing that: (i) dLLMs under parallel decoding can
suffer dramatic quality degradation in real-world scenarios, and (ii) current
parallel decoding strategies struggle to adapt their degree of parallelism
based on task difficulty, thus failing to achieve meaningful speedup without
compromising quality. Our findings underscore the pressing need for innovative
decoding methods that can overcome the current speed-quality trade-off. We
release our benchmark to help accelerate the development of truly efficient
dLLMs.
Parallel decoding in diffusion LLMs degrades generation quality due to ignored token dependencies, highlighting the need for new decoding methods and benchmarks.
autoregressive LLMs diffusion LLMs dLLMs parallel decoding conditional independence assumption information-theoretic analysis synthetic list operations ParallelBench real-world scenarios speed-quality trade-off