Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive
  Experts
Diffusion-based large language models (dLLMs) are trained flexibly to model
extreme dependence in the data distribution; however, how to best utilize this
information at inference time remains an open problem. In this work, we uncover
an interesting property of these models: dLLMs trained on textual data
implicitly learn a mixture of semi-autoregressive experts, where different
generation orders reveal different specialized behaviors. We show that
committing to any single, fixed inference time schedule, a common practice,
collapses performance by failing to leverage this latent ensemble. To address
this, we introduce HEX (Hidden semiautoregressive EXperts for test-time
scaling), a training-free inference method that ensembles across heterogeneous
block schedules. By doing a majority vote over diverse block-sized generation
paths, HEX robustly avoids failure modes associated with any single fixed
schedule. On reasoning benchmarks such as GSM8K, it boosts accuracy by up to
3.56X (from 24.72% to 88.10%), outperforming top-K margin inference and
specialized fine-tuned methods like GRPO, without additional training. HEX even
yields significant gains on MATH benchmark from 16.40% to 40.00%, scientific
reasoning on ARC-C from 54.18% to 87.80%, and TruthfulQA from 28.36% to 57.46%.
Our results establish a new paradigm for test-time scaling in diffusion-based
LLMs (dLLMs), revealing that the sequence in which masking is performed plays a
critical role in determining performance during inference.
68e4b83be4e093a7044e4e18
Jihoon Lee
68e4b83be4e093a7044e4e19
Hoyeon Moon
68e4b83be4e093a7044e4e1a
Kevin Zhai
68e4b83be4e093a7044e4e1b
Arun Kumar Chithanar
68e4b83be4e093a7044e4e1c
Anit Kumar Sahu
68e4b83be4e093a7044e4e1d
Soummya Kar
68e4b83be4e093a7044e4e1e
Chul Lee
68e4b83be4e093a7044e4e1f
Souradip Chakraborty
68e4b83be4e093a7044e4e20
Amrit Singh Bedi
claimed_verified
2025-10-07T12:26:08.097Z
68b905f152d028b4e78244a0
/avatars/e3b3304529b7d5c6362fb1324f92e285.svg
Amrit Singh Bedi
user
amrit0714
Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive
  Experts
Diffusion-based large language models (dLLMs) are trained flexibly to model
extreme dependence in the data distribution; however, how to best utilize this
information at inference time remains an open problem. In this work, we uncover
an interesting property of these models: dLLMs trained on textual data
implicitly learn a mixture of semi-autoregressive experts, where different
generation orders reveal different specialized behaviors. We show that
committing to any single, fixed inference time schedule, a common practice,
collapses performance by failing to leverage this latent ensemble. To address
this, we introduce HEX (Hidden semiautoregressive EXperts for test-time
scaling), a training-free inference method that ensembles across heterogeneous
block schedules. By doing a majority vote over diverse block-sized generation
paths, HEX robustly avoids failure modes associated with any single fixed
schedule. On reasoning benchmarks such as GSM8K, it boosts accuracy by up to
3.56X (from 24.72% to 88.10%), outperforming top-K margin inference and
specialized fine-tuned methods like GRPO, without additional training. HEX even
yields significant gains on MATH benchmark from 16.40% to 40.00%, scientific
reasoning on ARC-C from 54.18% to 87.80%, and TruthfulQA from 28.36% to 57.46%.
Our results establish a new paradigm for test-time scaling in diffusion-based
LLMs (dLLMs), revealing that the sequence in which masking is performed plays a
critical role in determining performance during inference.
HEX, a training-free inference method for diffusion-based large language models, ensembles diverse generation paths to improve accuracy across various reasoning benchmarks without additional training.
diffusion-based large language models dLLMs semi-autoregressive experts inference time schedule HEX Hidden semiautoregressive EXperts test-time scaling GSM8K MATH ARC-C TruthfulQA