Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
Discrete diffusion models offer a promising alternative to autoregressive
generation through parallel decoding, but they suffer from a sampling wall:
once categorical sampling occurs, rich distributional information collapses
into one-hot vectors and cannot be propagated across steps, forcing subsequent
steps to operate with limited information. To mitigate this problem, we
introduce Loopholing, a novel and simple mechanism that preserves this
information via a deterministic latent pathway, leading to Loopholing Discrete
Diffusion Models (LDDMs). Trained efficiently with a self-conditioning
strategy, LDDMs achieve substantial gains-reducing generative perplexity by up
to 61% over prior baselines, closing (and in some cases surpassing) the gap
with autoregressive models, and producing more coherent text. Applied to
reasoning tasks, LDDMs also improve performance on arithmetic benchmarks such
as Countdown and Game of 24. These results also indicate that loopholing
mitigates idle steps and oscillations, providing a scalable path toward
high-quality non-autoregressive text generation.
68fa31fef158a71c5a2f5608
Mingyu Jo
claimed_verified
2025-10-24T01:07:20.272Z
64c37ee87d89024360937d81
/avatars/cc9733b0862bbdca5e00f61a7ff7bb94.svg
mingyu jo
user
jojo0217
68fa31fef158a71c5a2f5609
Jaesik Yoon
68fa31fef158a71c5a2f560a
Justin Deschenaux
claimed_verified
2025-10-24T16:12:55.094Z
62d073f2485856cd710ed9fd
https://cdn-avatars.huggingface.co/v1/production/uploads/62d073f2485856cd710ed9fd/B6dck3fi5X9hhN4ALulMH.jpeg
Justin Deschenaux
user
jdeschena
68fa31fef158a71c5a2f560b
Caglar Gulcehre
68fa31fef158a71c5a2f560c
Sungjin Ahn
Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
Discrete diffusion models offer a promising alternative to autoregressive
generation through parallel decoding, but they suffer from a sampling wall:
once categorical sampling occurs, rich distributional information collapses
into one-hot vectors and cannot be propagated across steps, forcing subsequent
steps to operate with limited information. To mitigate this problem, we
introduce Loopholing, a novel and simple mechanism that preserves this
information via a deterministic latent pathway, leading to Loopholing Discrete
Diffusion Models (LDDMs). Trained efficiently with a self-conditioning
strategy, LDDMs achieve substantial gains-reducing generative perplexity by up
to 61% over prior baselines, closing (and in some cases surpassing) the gap
with autoregressive models, and producing more coherent text. Applied to
reasoning tasks, LDDMs also improve performance on arithmetic benchmarks such
as Countdown and Game of 24. These results also indicate that loopholing
mitigates idle steps and oscillations, providing a scalable path toward
high-quality non-autoregressive text generation.
Loopholing Discrete Diffusion Models (LDDMs) enhance text generation by preserving distributional information through a deterministic latent pathway, reducing perplexity and improving coherence and performance on reasoning tasks.
discrete diffusion models parallel decoding sampling wall one-hot vectors Loopholing deterministic latent pathway Loopholing Discrete Diffusion Models (LDDMs) self-conditioning strategy generative perplexity autoregressive models coherent text arithmetic benchmarks Countdown Game of 24 idle steps oscillations non-autoregressive text generation