CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers
  Up
Diffusion Transformers (DiT) have become a leading architecture in image
generation. However, the quadratic complexity of attention mechanisms, which
are responsible for modeling token-wise relationships, results in significant
latency when generating high-resolution images. To address this issue, we aim
at a linear attention mechanism in this paper that reduces the complexity of
pre-trained DiTs to linear. We begin our exploration with a comprehensive
summary of existing efficient attention mechanisms and identify four key
factors crucial for successful linearization of pre-trained DiTs: locality,
formulation consistency, high-rank attention maps, and feature integrity. Based
on these insights, we introduce a convolution-like local attention strategy
termed CLEAR, which limits feature interactions to a local window around each
query token, and thus achieves linear complexity. Our experiments indicate
that, by fine-tuning the attention layer on merely 10K self-generated samples
for 10K iterations, we can effectively transfer knowledge from a pre-trained
DiT to a student model with linear complexity, yielding results comparable to
the teacher model. Simultaneously, it reduces attention computations by 99.5%
and accelerates generation by 6.3 times for generating 8K-resolution images.
Furthermore, we investigate favorable properties in the distilled attention
layers, such as zero-shot generalization cross various models and plugins, and
improved support for multi-GPU parallel inference. Models and codes are
available here: https://github.com/Huage001/CLEAR.
6768d7f797a8f966b3362aa6
Songhua Liu
claimed_verified
2025-02-18T16:41:34.617Z
64b929308b53fb5dbd059ce3
/avatars/564c44cf45db794747a96f79f30ecd91.svg
Liu Songhua
user
Huage001
6768d7f797a8f966b3362aa7
Zhenxiong Tan
admin_assigned
2024-12-23T11:32:44.224Z
674e743be91289226ef9e857
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/FnLwJTTeItGbAUg4IPr7l.jpeg
唐振雄
user
ZhenxiongTang
6768d7f797a8f966b3362aa8
Xinchao Wang
admin_assigned
2024-12-23T11:33:21.470Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers
  Up
Diffusion Transformers (DiT) have become a leading architecture in image
generation. However, the quadratic complexity of attention mechanisms, which
are responsible for modeling token-wise relationships, results in significant
latency when generating high-resolution images. To address this issue, we aim
at a linear attention mechanism in this paper that reduces the complexity of
pre-trained DiTs to linear. We begin our exploration with a comprehensive
summary of existing efficient attention mechanisms and identify four key
factors crucial for successful linearization of pre-trained DiTs: locality,
formulation consistency, high-rank attention maps, and feature integrity. Based
on these insights, we introduce a convolution-like local attention strategy
termed CLEAR, which limits feature interactions to a local window around each
query token, and thus achieves linear complexity. Our experiments indicate
that, by fine-tuning the attention layer on merely 10K self-generated samples
for 10K iterations, we can effectively transfer knowledge from a pre-trained
DiT to a student model with linear complexity, yielding results comparable to
the teacher model. Simultaneously, it reduces attention computations by 99.5%
and accelerates generation by 6.3 times for generating 8K-resolution images.
Furthermore, we investigate favorable properties in the distilled attention
layers, such as zero-shot generalization cross various models and plugins, and
improved support for multi-GPU parallel inference. Models and codes are
available here: https://github.com/Huage001/CLEAR.
A convolution-like local attention strategy named CLEAR is introduced to reduce the quadratic complexity of Diffusion Transformers (DiT) to linear, significantly accelerating high-resolution image generation.
Diffusion Transformers attention mechanisms quadratic complexity token-wise relationships linear attention LATENT space locality formulation consistency high-rank attention maps feature integrity CLEAR self-generated samples zero-shot generalization multi-GPU parallel inference