Reconstruction vs. Generation: Taming Optimization Dilemma in Latent
  Diffusion Models
Latent diffusion models with Transformer architectures excel at generating
high-fidelity images. However, recent studies reveal an optimization dilemma in
this two-stage design: while increasing the per-token feature dimension in
visual tokenizers improves reconstruction quality, it requires substantially
larger diffusion models and more training iterations to achieve comparable
generation performance. Consequently, existing systems often settle for
sub-optimal solutions, either producing visual artifacts due to information
loss within tokenizers or failing to converge fully due to expensive
computation costs. We argue that this dilemma stems from the inherent
difficulty in learning unconstrained high-dimensional latent spaces. To address
this, we propose aligning the latent space with pre-trained vision foundation
models when training the visual tokenizers. Our proposed VA-VAE (Vision
foundation model Aligned Variational AutoEncoder) significantly expands the
reconstruction-generation frontier of latent diffusion models, enabling faster
convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces.
To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with
improved training strategies and architecture designs, termed LightningDiT. The
integrated system achieves state-of-the-art (SOTA) performance on ImageNet
256x256 generation with an FID score of 1.35 while demonstrating remarkable
training efficiency by reaching an FID score of 2.11 in just 64
epochs--representing an over 21 times convergence speedup compared to the
original DiT. Models and codes are available at:
https://github.com/hustvl/LightningDiT.
677753a28376dfe003a3fbd3
Jingfeng Yao
admin_assigned
2025-01-06T14:17:09.787Z
67756c9c846a267749304255
/avatars/01f09805b561887c55d1b9ad4e96b461.svg
Jingfeng Yao
user
MapleF9
677753a28376dfe003a3fbd4
Xinggang Wang
admin_assigned
2025-01-06T14:17:16.030Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
user
xinggangw
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent
  Diffusion Models
Latent diffusion models with Transformer architectures excel at generating
high-fidelity images. However, recent studies reveal an optimization dilemma in
this two-stage design: while increasing the per-token feature dimension in
visual tokenizers improves reconstruction quality, it requires substantially
larger diffusion models and more training iterations to achieve comparable
generation performance. Consequently, existing systems often settle for
sub-optimal solutions, either producing visual artifacts due to information
loss within tokenizers or failing to converge fully due to expensive
computation costs. We argue that this dilemma stems from the inherent
difficulty in learning unconstrained high-dimensional latent spaces. To address
this, we propose aligning the latent space with pre-trained vision foundation
models when training the visual tokenizers. Our proposed VA-VAE (Vision
foundation model Aligned Variational AutoEncoder) significantly expands the
reconstruction-generation frontier of latent diffusion models, enabling faster
convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces.
To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with
improved training strategies and architecture designs, termed LightningDiT. The
integrated system achieves state-of-the-art (SOTA) performance on ImageNet
256x256 generation with an FID score of 1.35 while demonstrating remarkable
training efficiency by reaching an FID score of 2.11 in just 64
epochs--representing an over 21 times convergence speedup compared to the
original DiT. Models and codes are available at:
https://github.com/hustvl/LightningDiT.
VA-VAE aligns latent spaces with pre-trained vision foundation models, enhancing DiT's training efficiency and performance on high-fidelity image generation.
latent diffusion models Transformer architectures visual tokenizers diffusion models pre-trained vision foundation models Variational AutoEncoder DiT ImageNet FID score LightningDiT