Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models
Donghoon Ahn, Jiwon Kang, Sanghyun Lee, Minjae Kim, Wooseok Jang, Jaewon Min, Sangwu Lee, Sayak Paul, Seungryong Kim
Recent guidance methods steer diffusion sampling by perturbing the model to construct an implicit bad model and guide generation away from it. Among them, attention perturbation has demonstrated strong empirical performance, yet existing approaches remain fragmented, necessitating a unified framework. We propose a generalized attention perturbation guidance framework that subsumes prior methods such as PAG and SEG as special cases, and show that interpolation between the original attention distribution and perturbed attention distribution effectively mitigates the oversimplification issue of existing attention perturbation guidance. As previously observed, only a small subset of layers contributes meaningfully to image quality improvements, making careful layer selection critical, especially in large-scale architectures such as DiT, where exhaustive perturbation evaluation is computationally prohibitive. To better understand how attention perturbations affect generation, we decompose the perturbation units into individual attention heads. Surprisingly, our per-head analysis reveals that specific heads govern distinct visual factors such as structure, lighting, or texture. Building on this, we introduce a systematic framework for head selection that allows for user-defined objectives, such as enhancing aesthetic quality and reinforcing specific visual styles. We validate our approach on modern large-scale text-to-image diffusion and flow models, including the Stable Diffusion 3 family and FLUX.1. Our work provides a unified view of attention perturbation guidance, uncovers interpretable structure within attention layers, and enables practical design of effective perturbation strategies in diffusion and flow models.