Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning
  Segmentation
Multimodal large language models (MLLMs) demonstrate strong video
understanding by attending to visual tokens relevant to textual queries. To
directly adapt this for localization in a training-free manner, we cast video
reasoning segmentation as a video QA task and extract attention maps via
rollout mechanism. However, raw attention maps are noisy and poorly aligned
with object regions. We propose Decomposed Attention Fusion (DecAF), which
refines these maps through two mechanisms: (1) contrastive object-background
fusion and (2) complementary video-frame fusion. This method suppresses
irrelevant activations and enhances object-focused cues, enabling direct
conversion of attention maps into coarse segmentation masks. In addition, we
introduce attention-guided SAM2 prompting for obtaining fine-grained masks.
Unlike existing methods that jointly train MLLMs with SAM, our method operates
entirely without retraining. DecAF outperforms training-free methods and
achieves performance comparable to training-based methods on both referring and
reasoning VOS benchmarks. The code will be available at
https://github.com/HYUNJS/DecAF.
68f9a201b9b2e4ae04673887
Su Ho Han
68f9a201b9b2e4ae04673888
Jeongseok Hyun
claimed_verified
2025-10-23T14:47:16.248Z
6513030fb3a463e17df56edd
/avatars/867bd4316b2de758654ad3a84ea868c1.svg
Hyun, Jeongseok
user
js-hyun
68f9a201b9b2e4ae04673889
Pilhyeon Lee
68f9a201b9b2e4ae0467388a
Minho Shim
68f9a201b9b2e4ae0467388b
Dongyoon Wee
68f9a201b9b2e4ae0467388c
Seon Joo Kim
Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning
  Segmentation
Multimodal large language models (MLLMs) demonstrate strong video
understanding by attending to visual tokens relevant to textual queries. To
directly adapt this for localization in a training-free manner, we cast video
reasoning segmentation as a video QA task and extract attention maps via
rollout mechanism. However, raw attention maps are noisy and poorly aligned
with object regions. We propose Decomposed Attention Fusion (DecAF), which
refines these maps through two mechanisms: (1) contrastive object-background
fusion and (2) complementary video-frame fusion. This method suppresses
irrelevant activations and enhances object-focused cues, enabling direct
conversion of attention maps into coarse segmentation masks. In addition, we
introduce attention-guided SAM2 prompting for obtaining fine-grained masks.
Unlike existing methods that jointly train MLLMs with SAM, our method operates
entirely without retraining. DecAF outperforms training-free methods and
achieves performance comparable to training-based methods on both referring and
reasoning VOS benchmarks. The code will be available at
https://github.com/HYUNJS/DecAF.
Decomposed Attention Fusion (DecAF) enhances video object segmentation by refining attention maps from multimodal large language models without retraining.
multimodal large language models video reasoning segmentation video QA task rollout mechanism attention maps contrastive object-background fusion complementary video-frame fusion attention-guided SAM2 prompting referring VOS benchmarks reasoning VOS benchmarks