BRAVE: Broadening the visual encoding of vision-language models
Vision-language models (VLMs) are typically composed of a vision encoder,
e.g. CLIP, and a language model (LM) that interprets the encoded features to
solve downstream tasks. Despite remarkable progress, VLMs are subject to
several shortcomings due to the limited capabilities of vision encoders, e.g.
"blindness" to certain image features, visual hallucination, etc. To address
these issues, we study broadening the visual encoding capabilities of VLMs. We
first comprehensively benchmark several vision encoders with different
inductive biases for solving VLM tasks. We observe that there is no single
encoding configuration that consistently achieves top performance across
different tasks, and encoders with different biases can perform surprisingly
similarly. Motivated by this, we introduce a method, named BRAVE, that
consolidates features from multiple frozen encoders into a more versatile
representation that can be directly fed as the input to a frozen LM. BRAVE
achieves state-of-the-art performance on a broad range of captioning and VQA
benchmarks and significantly reduces the aforementioned issues of VLMs, while
requiring a smaller number of trainable parameters than existing methods and
having a more compressed representation. Our results highlight the potential of
incorporating different visual biases for a more broad and contextualized
visual understanding of VLMs.
661740d051222b41dce25c80
Oğuzhan Fatih Kar
claimed_verified
2024-04-11T14:54:53.857Z
6303d05dd14428368d1866fd
/avatars/6c86567fac3cb253e0595aa678a2af03.svg
Oguzhan Fatih Kar
user
ozo
661740d051222b41dce25c81
Alessio Tonioni
admin_assigned
2024-04-18T13:27:49.848Z
63402b30e670ff9cf63d8caa
/avatars/0aee84d132a78d4ec71663836a57a245.svg
Alessio Tonioni
user
Alessiot
661740d051222b41dce25c82
Petra Poklukar
admin_assigned
2024-04-18T13:27:56.294Z
63f271b6f4e30ffd2bd4768b
/avatars/989c73fac91df3bc16b7a0460b97bc7b.svg
Petra Poklukar
user
petrapoklukar
661740d051222b41dce25c83
Achin Kulshrestha
661740d051222b41dce25c84
Amir Zamir
661740d051222b41dce25c85
Federico Tombari
BRAVE: Broadening the visual encoding of vision-language models
Vision-language models (VLMs) are typically composed of a vision encoder,
e.g. CLIP, and a language model (LM) that interprets the encoded features to
solve downstream tasks. Despite remarkable progress, VLMs are subject to
several shortcomings due to the limited capabilities of vision encoders, e.g.
"blindness" to certain image features, visual hallucination, etc. To address
these issues, we study broadening the visual encoding capabilities of VLMs. We
first comprehensively benchmark several vision encoders with different
inductive biases for solving VLM tasks. We observe that there is no single
encoding configuration that consistently achieves top performance across
different tasks, and encoders with different biases can perform surprisingly
similarly. Motivated by this, we introduce a method, named BRAVE, that
consolidates features from multiple frozen encoders into a more versatile
representation that can be directly fed as the input to a frozen LM. BRAVE
achieves state-of-the-art performance on a broad range of captioning and VQA
benchmarks and significantly reduces the aforementioned issues of VLMs, while
requiring a smaller number of trainable parameters than existing methods and
having a more compressed representation. Our results highlight the potential of
incorporating different visual biases for a more broad and contextualized
visual understanding of VLMs.
BRAVE consolidates features from multiple vision encoders to improve VLM performance on captioning and VQA tasks, addressing visual encoding limitations with reduced parameters and compressed representation.
vision-language models VLMs vision encoder CLIP language model LM visual encoding visual hallucination BRAVE inductive biases captioning VQA trainable parameters compressed representation visual understanding