Latent Diffusion Model without Variational Autoencoder
Recent progress in diffusion-based visual generation has largely relied on
latent diffusion models with variational autoencoders (VAEs). While effective
for high-fidelity synthesis, this VAE+diffusion paradigm suffers from limited
training efficiency, slow inference, and poor transferability to broader vision
tasks. These issues stem from a key limitation of VAE latent spaces: the lack
of clear semantic separation and strong discriminative structure. Our analysis
confirms that these properties are crucial not only for perception and
understanding tasks, but also for the stable and efficient training of latent
diffusion models. Motivated by this insight, we introduce SVG, a novel latent
diffusion model without variational autoencoders, which leverages
self-supervised representations for visual generation. SVG constructs a feature
space with clear semantic discriminability by leveraging frozen DINO features,
while a lightweight residual branch captures fine-grained details for
high-fidelity reconstruction. Diffusion models are trained directly on this
semantically structured latent space to facilitate more efficient learning. As
a result, SVG enables accelerated diffusion training, supports few-step
sampling, and improves generative quality. Experimental results further show
that SVG preserves the semantic and discriminative capabilities of the
underlying self-supervised representations, providing a principled pathway
toward task-general, high-quality visual representations.
68f59fc78589920bf4d32123
Minglei Shi
68f59fc78589920bf4d32124
Haolin Wang
claimed_verified
2025-10-20T12:10:58.245Z
641d373d353524fe41f1d453
/avatars/6fa9a0a4ba9818a221f835174a14be2d.svg
Haolin Wang
user
howlin
68f59fc78589920bf4d32125
Wenzhao Zheng
68f59fc78589920bf4d32126
Ziyang Yuan
68f59fc78589920bf4d32127
Xiaoshi Wu
68f59fc78589920bf4d32128
Xintao Wang
68f59fc78589920bf4d32129
Pengfei Wan
68f59fc78589920bf4d3212a
Jie Zhou
68f59fc78589920bf4d3212b
Jiwen Lu
Latent Diffusion Model without Variational Autoencoder
Recent progress in diffusion-based visual generation has largely relied on
latent diffusion models with variational autoencoders (VAEs). While effective
for high-fidelity synthesis, this VAE+diffusion paradigm suffers from limited
training efficiency, slow inference, and poor transferability to broader vision
tasks. These issues stem from a key limitation of VAE latent spaces: the lack
of clear semantic separation and strong discriminative structure. Our analysis
confirms that these properties are crucial not only for perception and
understanding tasks, but also for the stable and efficient training of latent
diffusion models. Motivated by this insight, we introduce SVG, a novel latent
diffusion model without variational autoencoders, which leverages
self-supervised representations for visual generation. SVG constructs a feature
space with clear semantic discriminability by leveraging frozen DINO features,
while a lightweight residual branch captures fine-grained details for
high-fidelity reconstruction. Diffusion models are trained directly on this
semantically structured latent space to facilitate more efficient learning. As
a result, SVG enables accelerated diffusion training, supports few-step
sampling, and improves generative quality. Experimental results further show
that SVG preserves the semantic and discriminative capabilities of the
underlying self-supervised representations, providing a principled pathway
toward task-general, high-quality visual representations.
SVG, a novel latent diffusion model without VAEs, uses self-supervised representations to enable efficient training, few-step sampling, and high-quality visual generation with semantic and discriminative capabilities.
latent diffusion models variational autoencoders diffusion models self-supervised representations DINO features residual branch high-fidelity reconstruction semantic discriminability task-general visual representations