DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
Diffusion models (DMs) have revolutionized generative learning. They utilize
a diffusion process to encode data into a simple Gaussian distribution.
However, encoding a complex, potentially multimodal data distribution into a
single continuous Gaussian distribution arguably represents an unnecessarily
challenging learning problem. We propose Discrete-Continuous Latent Variable
Diffusion Models (DisCo-Diff) to simplify this task by introducing
complementary discrete latent variables. We augment DMs with learnable discrete
latents, inferred with an encoder, and train DM and encoder end-to-end.
DisCo-Diff does not rely on pre-trained networks, making the framework
universally applicable. The discrete latents significantly simplify learning
the DM's complex noise-to-data mapping by reducing the curvature of the DM's
generative ODE. An additional autoregressive transformer models the
distribution of the discrete latents, a simple step because DisCo-Diff requires
only few discrete variables with small codebooks. We validate DisCo-Diff on toy
data, several image synthesis tasks as well as molecular docking, and find that
introducing discrete latents consistently improves model performance. For
example, DisCo-Diff achieves state-of-the-art FID scores on class-conditioned
ImageNet-64/128 datasets with ODE sampler.
6686497f1119f961e54211e9
Yilun Xu
admin_assigned
2024-07-04T12:35:48.060Z
649c37de5ffe05267a105fe8
/avatars/4f262272e6222f879c6c0fedfa2e5861.svg
Yilun Xu
user
AaronXyl
6686497f1119f961e54211ea
Gabriele Corso
admin_assigned
2024-07-04T12:46:42.221Z
63445a3332ccc5ca9938acd4
/avatars/4a1f6dd371094ada3addc10746a91616.svg
Gabriele Corso
user
gcorso
6686497f1119f961e54211eb
Tommi Jaakkola
6686497f1119f961e54211ec
Arash Vahdat
admin_assigned
2024-07-04T12:47:07.441Z
62f6956cd3bdacb7eec02920
/avatars/b22db0823311f866c00db2efc4b9f814.svg
Arash Vahdat
user
avahdat
6686497f1119f961e54211ed
Karsten Kreis
DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
Diffusion models (DMs) have revolutionized generative learning. They utilize
a diffusion process to encode data into a simple Gaussian distribution.
However, encoding a complex, potentially multimodal data distribution into a
single continuous Gaussian distribution arguably represents an unnecessarily
challenging learning problem. We propose Discrete-Continuous Latent Variable
Diffusion Models (DisCo-Diff) to simplify this task by introducing
complementary discrete latent variables. We augment DMs with learnable discrete
latents, inferred with an encoder, and train DM and encoder end-to-end.
DisCo-Diff does not rely on pre-trained networks, making the framework
universally applicable. The discrete latents significantly simplify learning
the DM's complex noise-to-data mapping by reducing the curvature of the DM's
generative ODE. An additional autoregressive transformer models the
distribution of the discrete latents, a simple step because DisCo-Diff requires
only few discrete variables with small codebooks. We validate DisCo-Diff on toy
data, several image synthesis tasks as well as molecular docking, and find that
introducing discrete latents consistently improves model performance. For
example, DisCo-Diff achieves state-of-the-art FID scores on class-conditioned
ImageNet-64/128 datasets with ODE sampler.
Discrete-Continuous Latent Variable Diffusion Models simplify generative learning by introducing discrete latent variables, enhancing performance across various tasks.
diffusion models Discrete-Continuous Latent Variable Diffusion Models DisCo-Diff discrete latent variables learnable discrete latents encoder end-to-end training ODE autoregressive transformer FID scores ImageNet-64/128 molecular docking