More Context, Less Distraction: Visual Classification by Inferring and
  Conditioning on Contextual Attributes
CLIP, as a foundational vision language model, is widely used in zero-shot
image classification due to its ability to understand various visual concepts
and natural language descriptions. However, how to fully leverage CLIP's
unprecedented human-like understanding capabilities to achieve better zero-shot
classification is still an open question. This paper draws inspiration from the
human visual perception process: a modern neuroscience view suggests that in
classifying an object, humans first infer its class-independent attributes
(e.g., background and orientation) which help separate the foreground object
from the background, and then make decisions based on this information.
Inspired by this, we observe that providing CLIP with contextual attributes
improves zero-shot classification and mitigates reliance on spurious features.
We also observe that CLIP itself can reasonably infer the attributes from an
image. With these observations, we propose a training-free, two-step zero-shot
classification method named PerceptionCLIP. Given an image, it first infers
contextual attributes (e.g., background) and then performs object
classification conditioning on them. Our experiments show that PerceptionCLIP
achieves better generalization, group robustness, and better interpretability.
For example, PerceptionCLIP with ViT-L/14 improves the worst group accuracy by
16.5% on the Waterbirds dataset and by 3.5% on CelebA.
64cb08617a4f236357164def
Bang An
extracted_confirmed
2023-08-03T02:30:29.418Z
645071555fdcff143ad8e07f
/avatars/a59f4e518a500e1c58b3a6d688644432.svg
Bang An
user
bangan
64cb08617a4f236357164df0
Sicheng Zhu
extracted_confirmed
2023-08-03T03:18:58.400Z
64cac1fd8174e45ae03f12fa
/avatars/21a80c732160a9a594215cd7f012f619.svg
Sicheng Zhu
user
sichengzhu
64cb08617a4f236357164df1
Michael-Andrei Panaitescu-Liess
extracted_pending
2024-01-21T05:26:18.694Z
65acaaa3b68db4f26e9e71d5
/avatars/41da4b51d212d84c2b0eb8fca2eea477.svg
Michael-Andrei Panaitescu-Liess
user
michaelpanaitescu
64cb08617a4f236357164df2
Chaithanya Kumar Mummadi
64cb08617a4f236357164df3
Furong Huang
extracted_confirmed
2023-08-03T15:19:48.595Z
64cbc3e2a257a3212c00a115
/avatars/836e61be4aeda2080ddf2db9f2626cc6.svg
Furong Huang Lab at UMD
user
furongh-lab
More Context, Less Distraction: Visual Classification by Inferring and
  Conditioning on Contextual Attributes
CLIP, as a foundational vision language model, is widely used in zero-shot
image classification due to its ability to understand various visual concepts
and natural language descriptions. However, how to fully leverage CLIP's
unprecedented human-like understanding capabilities to achieve better zero-shot
classification is still an open question. This paper draws inspiration from the
human visual perception process: a modern neuroscience view suggests that in
classifying an object, humans first infer its class-independent attributes
(e.g., background and orientation) which help separate the foreground object
from the background, and then make decisions based on this information.
Inspired by this, we observe that providing CLIP with contextual attributes
improves zero-shot classification and mitigates reliance on spurious features.
We also observe that CLIP itself can reasonably infer the attributes from an
image. With these observations, we propose a training-free, two-step zero-shot
classification method named PerceptionCLIP. Given an image, it first infers
contextual attributes (e.g., background) and then performs object
classification conditioning on them. Our experiments show that PerceptionCLIP
achieves better generalization, group robustness, and better interpretability.
For example, PerceptionCLIP with ViT-L/14 improves the worst group accuracy by
16.5% on the Waterbirds dataset and by 3.5% on CelebA.
A two-step zero-shot classification method, named PerceptionCLIP, improves CLIP's performance by inferring contextual attributes from images before classification.
CLIP zero-shot image classification human visual perception class-independent attributes contextual attributes attribute inference PerceptionCLIP group robustness interpretability ViT-L/14 Waterbirds dataset CelebA