Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens
  for Text-to-Any-Task
The global self-attention mechanism in diffusion transformers involves
redundant computation due to the sparse and redundant nature of visual
information, and the attention map of tokens within a spatial window shows
significant similarity. To address this redundancy, we propose the Proxy Token
Diffusion Transformer (PT-DiT), which employs sparse representative token
attention (where the number of representative tokens is much smaller than the
total number of tokens) to model global visual information efficiently.
Specifically, in each transformer block, we randomly sample one token from each
spatial-temporal window to serve as a proxy token for that region. The global
semantics are captured through the self-attention of these proxy tokens and
then injected into all latent tokens via cross-attention. Simultaneously, we
introduce window and shift window attention to address the limitations in
detail modeling caused by the sparse attention mechanism. Building on the
well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a
variety of models for T2I, T2V, and T2MV tasks. Experimental results show that
PT-DiT achieves competitive performance while reducing the computational
complexity in both image and video generation tasks (e.g., a 48% reduction
compared to DiT and a 35% reduction compared to Pixart-alpha). Our source code
is available at https://github.com/360CVGroup/Qihoo-T2X.
66de667dfd7d68a2932da0c6
Jing Wang
66de667dfd7d68a2932da0c7
Ao Ma
admin_assigned
2024-09-09T08:02:08.111Z
65292034825f4bba46c78581
https://cdn-avatars.huggingface.co/v1/production/uploads/65292034825f4bba46c78581/8pkETWFhha7AUM6E37Rvl.jpeg
AoMa
user
AoMa
66de667dfd7d68a2932da0c8
Jiasong Feng
claimed_verified
2024-09-10T08:37:59.995Z
65542ae4520d9bb9d9d21659
/avatars/c89f2e76f2f1f4e162d15f4da8280f19.svg
js
user
feng8
66de667dfd7d68a2932da0c9
Dawei Leng
extracted_confirmed
2024-10-18T02:23:14.520Z
649935abbe8fd92c27ab1ed8
https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
David Leon
user
DavidLeon
66de667dfd7d68a2932da0ca
Yuhui Yin
66de667dfd7d68a2932da0cb
Xiaodan Liang
Qihoo-T2X: An Efficiency-Focused Diffusion Transformer via Proxy Tokens
  for Text-to-Any-Task
The global self-attention mechanism in diffusion transformers involves
redundant computation due to the sparse and redundant nature of visual
information, and the attention map of tokens within a spatial window shows
significant similarity. To address this redundancy, we propose the Proxy Token
Diffusion Transformer (PT-DiT), which employs sparse representative token
attention (where the number of representative tokens is much smaller than the
total number of tokens) to model global visual information efficiently.
Specifically, in each transformer block, we randomly sample one token from each
spatial-temporal window to serve as a proxy token for that region. The global
semantics are captured through the self-attention of these proxy tokens and
then injected into all latent tokens via cross-attention. Simultaneously, we
introduce window and shift window attention to address the limitations in
detail modeling caused by the sparse attention mechanism. Building on the
well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a
variety of models for T2I, T2V, and T2MV tasks. Experimental results show that
PT-DiT achieves competitive performance while reducing the computational
complexity in both image and video generation tasks (e.g., a 48% reduction
compared to DiT and a 35% reduction compared to Pixart-alpha). Our source code
is available at https://github.com/360CVGroup/Qihoo-T2X.
The Proxy Token Diffusion Transformer (PT-DiT) uses sparse representative token attention to efficiently capture global visual information and reduce redundancy in diffusion transformers, achieving competitive performance with reduced computational complexity.
global self-attention diffusion transformers sparse representative token attention proxy tokens cross-attention window attention shift window attention Qihoo-T2X T2I T2V T2MV tasks