Inherently Faithful Attention Maps for Vision Transformers
We introduce an attention-based method that uses learned binary attention
masks to ensure that only attended image regions influence the prediction.
Context can strongly affect object perception, sometimes leading to biased
representations, particularly when objects appear in out-of-distribution
backgrounds. At the same time, many image-level object-centric tasks require
identifying relevant regions, often requiring context. To address this
conundrum, we propose a two-stage framework: stage 1 processes the full image
to discover object parts and identify task-relevant regions, while stage 2
leverages input attention masking to restrict its receptive field to these
regions, enabling a focused analysis while filtering out potentially spurious
information. Both stages are trained jointly, allowing stage 2 to refine stage
1. Extensive experiments across diverse benchmarks demonstrate that our
approach significantly improves robustness against spurious correlations and
out-of-distribution backgrounds.
684fe3711d9b438aa39579da
Ananthu Aniraj
claimed_verified
2025-06-16T09:52:38.316Z
6508647f0c87331947c4a46d
/avatars/9ddaf4ec53729cb69f65b314a7f4a9a0.svg
Ananthu Aniraj
user
ananthu-aniraj
684fe3711d9b438aa39579db
Cassio F. Dantas
claimed_verified
2025-06-16T12:56:17.943Z
6642254468b89fe3453439f1
https://cdn-avatars.huggingface.co/v1/production/uploads/6642254468b89fe3453439f1/9lgQXPykZ5XYiwCQsE8ia.jpeg
Cassio F. Dantas
user
CassioFraga
684fe3711d9b438aa39579dc
Dino Ienco
684fe3711d9b438aa39579dd
Diego Marcos
Inherently Faithful Attention Maps for Vision Transformers
We introduce an attention-based method that uses learned binary attention
masks to ensure that only attended image regions influence the prediction.
Context can strongly affect object perception, sometimes leading to biased
representations, particularly when objects appear in out-of-distribution
backgrounds. At the same time, many image-level object-centric tasks require
identifying relevant regions, often requiring context. To address this
conundrum, we propose a two-stage framework: stage 1 processes the full image
to discover object parts and identify task-relevant regions, while stage 2
leverages input attention masking to restrict its receptive field to these
regions, enabling a focused analysis while filtering out potentially spurious
information. Both stages are trained jointly, allowing stage 2 to refine stage
1. Extensive experiments across diverse benchmarks demonstrate that our
approach significantly improves robustness against spurious correlations and
out-of-distribution backgrounds.
An attention-based method using learned binary masks improves robustness in object perception by focusing on relevant image regions while filtering out spurious information.
attention-based method learned binary attention masks object perception context out-of-distribution backgrounds image-level object-centric tasks task-relevant regions two-stage framework receptive field joint training robustness spurious correlations