Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
Vision-Language Models (VLMs) have demonstrated remarkable success across
diverse visual tasks, yet their performance degrades in complex visual
environments. While existing enhancement approaches require additional
training, rely on external segmentation tools, or operate at coarse-grained
levels, they overlook the innate ability within VLMs. To bridge this gap, we
investigate VLMs' attention patterns and discover that: (1) visual complexity
strongly correlates with attention entropy, negatively impacting reasoning
performance; (2) attention progressively refines from global scanning in
shallow layers to focused convergence in deeper layers, with convergence degree
determined by visual complexity. (3) Theoretically, we prove that the contrast
of attention maps between general queries and task-specific queries enables the
decomposition of visual signal into semantic signals and visual noise
components. Building on these insights, we propose Contrastive Attention
Refinement for Visual Enhancement (CARVE), a training-free method that extracts
task-relevant visual signals through attention contrasting at the pixel level.
Extensive experiments demonstrate that CARVE consistently enhances performance,
achieving up to 75% improvement on open-source models. Our work provides
critical insights into the interplay between visual complexity and attention
mechanisms, offering an efficient pathway for improving visual reasoning with
contrasting attention.
68bfa200207285de11b07beb
Yuyao Ge
claimed_verified
2025-09-09T13:47:32.746Z
656ad93853703dd78f3de7b8
https://cdn-avatars.huggingface.co/v1/production/uploads/656ad93853703dd78f3de7b8/r6VB6ICND1td3wFOy8pnz.jpeg
YuyaoGe
user
YuyaoGe
68bfa200207285de11b07bec
Shenghua Liu
68bfa200207285de11b07bed
Yiwei Wang
68bfa200207285de11b07bee
Lingrui Mei
68bfa200207285de11b07bef
Baolong Bi
68bfa200207285de11b07bf0
Xuanshan Zhou
68bfa200207285de11b07bf1
Jiayu Yao
68bfa200207285de11b07bf2
Jiafeng Guo
68bfa200207285de11b07bf3
Xueqi Cheng
Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
Vision-Language Models (VLMs) have demonstrated remarkable success across
diverse visual tasks, yet their performance degrades in complex visual
environments. While existing enhancement approaches require additional
training, rely on external segmentation tools, or operate at coarse-grained
levels, they overlook the innate ability within VLMs. To bridge this gap, we
investigate VLMs' attention patterns and discover that: (1) visual complexity
strongly correlates with attention entropy, negatively impacting reasoning
performance; (2) attention progressively refines from global scanning in
shallow layers to focused convergence in deeper layers, with convergence degree
determined by visual complexity. (3) Theoretically, we prove that the contrast
of attention maps between general queries and task-specific queries enables the
decomposition of visual signal into semantic signals and visual noise
components. Building on these insights, we propose Contrastive Attention
Refinement for Visual Enhancement (CARVE), a training-free method that extracts
task-relevant visual signals through attention contrasting at the pixel level.
Extensive experiments demonstrate that CARVE consistently enhances performance,
achieving up to 75% improvement on open-source models. Our work provides
critical insights into the interplay between visual complexity and attention
mechanisms, offering an efficient pathway for improving visual reasoning with
contrasting attention.
Contrastive Attention Refinement for Visual Enhancement (CARVE) improves VLM performance by extracting task-relevant visual signals through attention contrasting, addressing issues with visual complexity and attention mechanisms.
Vision-Language Models attention patterns attention entropy reasoning performance attention maps general queries task-specific queries semantic signals visual noise Contrastive Attention Refinement CARVE visual enhancement visual complexity attention mechanisms visual reasoning