Hidden in plain sight: VLMs overlook their visual representations
Language provides a natural interface to specify and evaluate performance on
visual tasks. To realize this possibility, vision language models (VLMs) must
successfully integrate visual and linguistic information. Our work compares
VLMs to a direct readout of their visual encoders to understand their ability
to integrate across these modalities. Across a series of vision-centric
benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform
substantially worse than their visual encoders, dropping to near-chance
performance. We investigate these results through a series of analyses across
the entire VLM: namely 1) the degradation of vision representations, 2)
brittleness to task prompt, and 3) the language model's role in solving the
task. We find that the bottleneck in performing these vision-centric tasks lies
in this third category; VLMs are not effectively using visual information
easily accessible throughout the entire model, and they inherit the language
priors present in the LLM. Our work helps diagnose the failure modes of
open-source VLMs, and presents a series of evaluations useful for future
investigations into visual understanding within VLMs.
684ae06cdbd21a9cc27b0ecf
Stephanie Fu
684ae06cdbd21a9cc27b0ed0
Tyler Bonnen
684ae06cdbd21a9cc27b0ed1
Devin Guillory
684ae06cdbd21a9cc27b0ed2
Trevor Darrell
Hidden in plain sight: VLMs overlook their visual representations
Language provides a natural interface to specify and evaluate performance on
visual tasks. To realize this possibility, vision language models (VLMs) must
successfully integrate visual and linguistic information. Our work compares
VLMs to a direct readout of their visual encoders to understand their ability
to integrate across these modalities. Across a series of vision-centric
benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform
substantially worse than their visual encoders, dropping to near-chance
performance. We investigate these results through a series of analyses across
the entire VLM: namely 1) the degradation of vision representations, 2)
brittleness to task prompt, and 3) the language model's role in solving the
task. We find that the bottleneck in performing these vision-centric tasks lies
in this third category; VLMs are not effectively using visual information
easily accessible throughout the entire model, and they inherit the language
priors present in the LLM. Our work helps diagnose the failure modes of
open-source VLMs, and presents a series of evaluations useful for future
investigations into visual understanding within VLMs.
Vision language models perform worse than their visual encoders on vision-centric tasks due to ineffective use of visual information and language priors.
vision language models visual encoders depth estimation correspondence vision representations task prompt language model open-source VLMs visual understanding