DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage
  Mitigation in Text-to-Image Models
Text-to-Image (T2I) models have advanced rapidly, yet they remain vulnerable
to semantic leakage, the unintended transfer of semantically related features
between distinct entities. Existing mitigation strategies are often
optimization-based or dependent on external inputs. We introduce DeLeaker, a
lightweight, optimization-free inference-time approach that mitigates leakage
by directly intervening on the model's attention maps. Throughout the diffusion
process, DeLeaker dynamically reweights attention maps to suppress excessive
cross-entity interactions while strengthening the identity of each entity. To
support systematic evaluation, we introduce SLIM (Semantic Leakage in IMages),
the first dataset dedicated to semantic leakage, comprising 1,130
human-verified samples spanning diverse scenarios, together with a novel
automatic evaluation framework. Experiments demonstrate that DeLeaker
consistently outperforms all baselines, even when they are provided with
external information, achieving effective leakage mitigation without
compromising fidelity or quality. These results underscore the value of
attention control and pave the way for more semantically precise T2I models.
68fa18a3f158a71c5a2f55cf
Mor Ventura
68fa18a3f158a71c5a2f55d0
Michael Toker
68fa18a3f158a71c5a2f55d1
Or Patashnik
68fa18a3f158a71c5a2f55d2
Yonatan Belinkov
68fa18a3f158a71c5a2f55d3
Roi Reichart
DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage
  Mitigation in Text-to-Image Models
Text-to-Image (T2I) models have advanced rapidly, yet they remain vulnerable
to semantic leakage, the unintended transfer of semantically related features
between distinct entities. Existing mitigation strategies are often
optimization-based or dependent on external inputs. We introduce DeLeaker, a
lightweight, optimization-free inference-time approach that mitigates leakage
by directly intervening on the model's attention maps. Throughout the diffusion
process, DeLeaker dynamically reweights attention maps to suppress excessive
cross-entity interactions while strengthening the identity of each entity. To
support systematic evaluation, we introduce SLIM (Semantic Leakage in IMages),
the first dataset dedicated to semantic leakage, comprising 1,130
human-verified samples spanning diverse scenarios, together with a novel
automatic evaluation framework. Experiments demonstrate that DeLeaker
consistently outperforms all baselines, even when they are provided with
external information, achieving effective leakage mitigation without
compromising fidelity or quality. These results underscore the value of
attention control and pave the way for more semantically precise T2I models.
DeLeaker mitigates semantic leakage in text-to-image models by dynamically reweighting attention maps during the diffusion process, outperforming existing methods without compromising quality.
text-to-image models semantic leakage attention maps diffusion process SLIM automatic evaluation framework