Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion
  Models
This study explores the role of cross-attention during inference in
text-conditional diffusion models. We find that cross-attention outputs
converge to a fixed point after few inference steps. Accordingly, the time
point of convergence naturally divides the entire inference process into two
stages: an initial semantics-planning stage, during which, the model relies on
cross-attention to plan text-oriented visual semantics, and a subsequent
fidelity-improving stage, during which the model tries to generate images from
previously planned semantics. Surprisingly, ignoring text conditions in the
fidelity-improving stage not only reduces computation complexity, but also
maintains model performance. This yields a simple and training-free method
called TGATE for efficient generation, which caches the cross-attention output
once it converges and keeps it fixed during the remaining inference steps. Our
empirical study on the MS-COCO validation set confirms its effectiveness. The
source code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.
660e1a88e8763d8d1da50fa6
Wentian Zhang
660e1a88e8763d8d1da50fa7
Haozhe Liu
660e1a88e8763d8d1da50fa8
Jinheng Xie
claimed_verified
2025-06-19T09:12:17.746Z
64731de50da364bd0dfb7cd0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/6DTmh7MOemA6xYMWGY9Ce.jpeg
Jinheng Xie
user
Sierkinhane
660e1a88e8763d8d1da50fa9
Francesco Faccio
660e1a88e8763d8d1da50faa
Mike Zheng Shou
660e1a88e8763d8d1da50fab
Jürgen Schmidhuber
Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion
  Models
This study explores the role of cross-attention during inference in
text-conditional diffusion models. We find that cross-attention outputs
converge to a fixed point after few inference steps. Accordingly, the time
point of convergence naturally divides the entire inference process into two
stages: an initial semantics-planning stage, during which, the model relies on
cross-attention to plan text-oriented visual semantics, and a subsequent
fidelity-improving stage, during which the model tries to generate images from
previously planned semantics. Surprisingly, ignoring text conditions in the
fidelity-improving stage not only reduces computation complexity, but also
maintains model performance. This yields a simple and training-free method
called TGATE for efficient generation, which caches the cross-attention output
once it converges and keeps it fixed during the remaining inference steps. Our
empirical study on the MS-COCO validation set confirms its effectiveness. The
source code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.
Cross-attention in text-conditional diffusion models divides inference into semantics-planning and fidelity-improving stages, leading to an efficient generation method called TGATE.
cross-attention diffusion models inference semantics-planning stage fidelity-improving stage TGATE MS-COCO