D^2iT: Dynamic Diffusion Transformer for Accurate Image Generation
Diffusion models are widely recognized for their ability to generate
high-fidelity images. Despite the excellent performance and scalability of the
Diffusion Transformer (DiT) architecture, it applies fixed compression across
different image regions during the diffusion process, disregarding the
naturally varying information densities present in these regions. However,
large compression leads to limited local realism, while small compression
increases computational complexity and compromises global consistency,
ultimately impacting the quality of generated images. To address these
limitations, we propose dynamically compressing different image regions by
recognizing the importance of different regions, and introduce a novel
two-stage framework designed to enhance the effectiveness and efficiency of
image generation: (1) Dynamic VAE (DVAE) at first stage employs a hierarchical
encoder to encode different image regions at different downsampling rates,
tailored to their specific information densities, thereby providing more
accurate and natural latent codes for the diffusion process. (2) Dynamic
Diffusion Transformer (D^2iT) at second stage generates images by predicting
multi-grained noise, consisting of coarse-grained (less latent code in smooth
regions) and fine-grained (more latent codes in detailed regions), through an
novel combination of the Dynamic Grain Transformer and the Dynamic Content
Transformer. The strategy of combining rough prediction of noise with detailed
regions correction achieves a unification of global consistency and local
realism. Comprehensive experiments on various generation tasks validate the
effectiveness of our approach. Code will be released at
https://github.com/jiawn-creator/Dynamic-DiT.
67ff6f8c661b74d0050d2774
Weinan Jia
67ff6f8c661b74d0050d2775
Mengqi Huang
67ff6f8c661b74d0050d2776
Nan Chen
claimed_verified
2025-06-03T08:51:48.179Z
6629d7c9fa14eaccf07d8633
/avatars/dceb2f6c804c583adf15a3536c8c995b.svg
Nan Chen
user
CNcreator0331
67ff6f8c661b74d0050d2777
Lei Zhang
67ff6f8c661b74d0050d2778
Zhendong Mao
D^2iT: Dynamic Diffusion Transformer for Accurate Image Generation
Diffusion models are widely recognized for their ability to generate
high-fidelity images. Despite the excellent performance and scalability of the
Diffusion Transformer (DiT) architecture, it applies fixed compression across
different image regions during the diffusion process, disregarding the
naturally varying information densities present in these regions. However,
large compression leads to limited local realism, while small compression
increases computational complexity and compromises global consistency,
ultimately impacting the quality of generated images. To address these
limitations, we propose dynamically compressing different image regions by
recognizing the importance of different regions, and introduce a novel
two-stage framework designed to enhance the effectiveness and efficiency of
image generation: (1) Dynamic VAE (DVAE) at first stage employs a hierarchical
encoder to encode different image regions at different downsampling rates,
tailored to their specific information densities, thereby providing more
accurate and natural latent codes for the diffusion process. (2) Dynamic
Diffusion Transformer (D^2iT) at second stage generates images by predicting
multi-grained noise, consisting of coarse-grained (less latent code in smooth
regions) and fine-grained (more latent codes in detailed regions), through an
novel combination of the Dynamic Grain Transformer and the Dynamic Content
Transformer. The strategy of combining rough prediction of noise with detailed
regions correction achieves a unification of global consistency and local
realism. Comprehensive experiments on various generation tasks validate the
effectiveness of our approach. Code will be released at
https://github.com/jiawn-creator/Dynamic-DiT.
A two-stage framework combining dynamic VAE encoding and dynamic diffusion transformer generation enhances image quality by adapting compression to regional information density.
diffusion models Diffusion Transformer Dynamic VAE hierarchical encoder downsampling rates information densities latent codes Dynamic Diffusion Transformer Dynamic Grain Transformer Dynamic Content Transformer multi-grained noise coarse-grained fine-grained global consistency local realism