Continuously Augmented Discrete Diffusion model for Categorical
  Generative Modeling
Standard discrete diffusion models treat all unobserved states identically by
mapping them to an absorbing [MASK] token. This creates an 'information void'
where semantic information that could be inferred from unmasked tokens is lost
between denoising steps. We introduce Continuously Augmented Discrete Diffusion
(CADD), a framework that augments the discrete state space with a paired
diffusion in a continuous latent space. This yields graded, gradually corrupted
states in which masked tokens are represented by noisy yet informative latent
vectors rather than collapsed 'information voids'. At each reverse step, CADD
may leverage the continuous latent as a semantic hint to guide discrete
denoising. The design is clean and compatible with existing discrete diffusion
training. At sampling time, the strength and choice of estimator for the
continuous latent vector enables a controlled trade-off between mode-coverage
(generating diverse outputs) and mode-seeking (generating contextually precise
outputs) behaviors. Empirically, we demonstrate CADD improves generative
quality over mask-based diffusion across text generation, image synthesis, and
code modeling, with consistent gains on both qualitative and quantitative
metrics against strong discrete baselines.
68e3e14773e20ab577842284
Huangjie Zheng
68e3e14773e20ab577842285
Shansan Gong
68e3e14773e20ab577842286
Ruixiang Zhang
68e3e14773e20ab577842287
Tianrong Chen
68e3e14773e20ab577842288
Jiatao Gu
68e3e14773e20ab577842289
Mingyuan Zhou
68e3e14773e20ab57784228a
Navdeep Jaitly
68e3e14773e20ab57784228b
Yizhe Zhang
Continuously Augmented Discrete Diffusion model for Categorical
  Generative Modeling
Standard discrete diffusion models treat all unobserved states identically by
mapping them to an absorbing [MASK] token. This creates an 'information void'
where semantic information that could be inferred from unmasked tokens is lost
between denoising steps. We introduce Continuously Augmented Discrete Diffusion
(CADD), a framework that augments the discrete state space with a paired
diffusion in a continuous latent space. This yields graded, gradually corrupted
states in which masked tokens are represented by noisy yet informative latent
vectors rather than collapsed 'information voids'. At each reverse step, CADD
may leverage the continuous latent as a semantic hint to guide discrete
denoising. The design is clean and compatible with existing discrete diffusion
training. At sampling time, the strength and choice of estimator for the
continuous latent vector enables a controlled trade-off between mode-coverage
(generating diverse outputs) and mode-seeking (generating contextually precise
outputs) behaviors. Empirically, we demonstrate CADD improves generative
quality over mask-based diffusion across text generation, image synthesis, and
code modeling, with consistent gains on both qualitative and quantitative
metrics against strong discrete baselines.
Continuously Augmented Discrete Diffusion (CADD) enhances generative quality by integrating a continuous latent space into discrete diffusion models, providing informative latent vectors for masked tokens and improving mode-coverage and mode-seeking behaviors.
discrete diffusion models absorbing token information void continuous latent space graded gradually corrupted states discrete denoising semantic hint mode-coverage mode-seeking text generation image synthesis code modeling