RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Most existing approaches to referring segmentation achieve strong performance
only through fine-tuning or by composing multiple pre-trained models, often at
the cost of additional training and architectural modifications. Meanwhile,
large-scale generative diffusion models encode rich semantic information,
making them attractive as general-purpose feature extractors. In this work, we
introduce a new method that directly exploits features, attention scores, from
diffusion transformers for downstream tasks, requiring neither architectural
modifications nor additional training. To systematically evaluate these
features, we extend benchmarks with vision-language grounding tasks spanning
both images and videos. Our key insight is that stop words act as attention
magnets: they accumulate surplus attention and can be filtered to reduce noise.
Moreover, we identify global attention sinks (GAS) emerging in deeper layers
and show that they can be safely suppressed or redirected onto auxiliary
tokens, leading to sharper and more accurate grounding maps. We further propose
an attention redistribution strategy, where appended stop words partition
background activations into smaller clusters, yielding sharper and more
localized heatmaps. Building on these findings, we develop RefAM, a simple
training-free grounding framework that combines cross-attention maps, GAS
handling, and redistribution. Across zero-shot referring image and video
segmentation benchmarks, our approach consistently outperforms prior methods,
establishing a new state of the art without fine-tuning or additional
components.
68da30c00177a6054b013c28
Anna Kukleva
68da30c00177a6054b013c29
Enis Simsar
claimed_verified
2025-09-29T11:26:19.128Z
63412f2add8853dc7e306a4f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nqbteeQhS2dPA5LVy7dTD.png
Enis Simsar
user
enisimsar
68da30c00177a6054b013c2a
Alessio Tonioni
68da30c00177a6054b013c2b
Muhammad Ferjad Naeem
68da30c00177a6054b013c2c
Federico Tombari
68da30c00177a6054b013c2d
Jan Eric Lenssen
68da30c00177a6054b013c2e
Bernt Schiele
RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Most existing approaches to referring segmentation achieve strong performance
only through fine-tuning or by composing multiple pre-trained models, often at
the cost of additional training and architectural modifications. Meanwhile,
large-scale generative diffusion models encode rich semantic information,
making them attractive as general-purpose feature extractors. In this work, we
introduce a new method that directly exploits features, attention scores, from
diffusion transformers for downstream tasks, requiring neither architectural
modifications nor additional training. To systematically evaluate these
features, we extend benchmarks with vision-language grounding tasks spanning
both images and videos. Our key insight is that stop words act as attention
magnets: they accumulate surplus attention and can be filtered to reduce noise.
Moreover, we identify global attention sinks (GAS) emerging in deeper layers
and show that they can be safely suppressed or redirected onto auxiliary
tokens, leading to sharper and more accurate grounding maps. We further propose
an attention redistribution strategy, where appended stop words partition
background activations into smaller clusters, yielding sharper and more
localized heatmaps. Building on these findings, we develop RefAM, a simple
training-free grounding framework that combines cross-attention maps, GAS
handling, and redistribution. Across zero-shot referring image and video
segmentation benchmarks, our approach consistently outperforms prior methods,
establishing a new state of the art without fine-tuning or additional
components.
A new method leverages diffusion transformers' attention scores for referring segmentation without fine-tuning or additional training, improving performance through stop word filtering and attention redistribution.
diffusion transformers attention scores vision-language grounding global attention sinks attention redistribution cross-attention maps RefAM zero-shot referring image and video segmentation