Lost in Embeddings: Information Loss in Vision-Language Models
Vision--language models (VLMs) often process visual inputs through a
pretrained vision encoder, followed by a projection into the language model's
embedding space via a connector component. While crucial for modality fusion,
the potential information loss induced by this projection step and its direct
impact on model capabilities remain understudied. We introduce two
complementary approaches to examine and quantify this loss by analyzing the
latent representation space. First, we evaluate semantic information
preservation by analyzing changes in k-nearest neighbor relationships between
image representations, before and after projection. Second, we directly measure
information loss by reconstructing visual embeddings from the projected
representation, localizing loss at an image patch level. Experiments reveal
that connectors substantially distort the local geometry of visual
representations, with k-nearest neighbors diverging by 40--60\%
post-projection, correlating with degradation in retrieval performance. The
patch-level embedding reconstruction provides interpretable insights for model
behavior on visually grounded question-answering tasks, finding that areas of
high information loss reliably predict instances where models struggle.
68c8d7dd733e345e52ac1edd
Wenyan Li
claimed_verified
2025-09-16T09:41:46.176Z
619b506f70d03780cbec5806
/avatars/ea2b0b8f0a3eb16d53ef40da9981c397.svg
wenyan li
user
lyan62
68c8d7dd733e345e52ac1ede
Raphael Tang
claimed_verified
2025-09-22T02:17:51.437Z
63250bb8d206fe7b2d2f1b8b
https://cdn-avatars.huggingface.co/v1/production/uploads/1668053024087-63250bb8d206fe7b2d2f1b8b.jpeg
Raphael Tang
user
tetrisd
68c8d7dd733e345e52ac1edf
Chengzu Li
68c8d7dd733e345e52ac1ee0
Caiqi Zhang
68c8d7dd733e345e52ac1ee1
Ivan Vulić
68c8d7dd733e345e52ac1ee2
Anders Søgaard
Lost in Embeddings: Information Loss in Vision-Language Models
Vision--language models (VLMs) often process visual inputs through a
pretrained vision encoder, followed by a projection into the language model's
embedding space via a connector component. While crucial for modality fusion,
the potential information loss induced by this projection step and its direct
impact on model capabilities remain understudied. We introduce two
complementary approaches to examine and quantify this loss by analyzing the
latent representation space. First, we evaluate semantic information
preservation by analyzing changes in k-nearest neighbor relationships between
image representations, before and after projection. Second, we directly measure
information loss by reconstructing visual embeddings from the projected
representation, localizing loss at an image patch level. Experiments reveal
that connectors substantially distort the local geometry of visual
representations, with k-nearest neighbors diverging by 40--60\%
post-projection, correlating with degradation in retrieval performance. The
patch-level embedding reconstruction provides interpretable insights for model
behavior on visually grounded question-answering tasks, finding that areas of
high information loss reliably predict instances where models struggle.
Two approaches are introduced to analyze and quantify information loss in vision-language models during the projection of visual inputs into the language model's embedding space, revealing significant distortions and their impact on model performance.
vision--language models pretrained vision encoder connector component modality fusion latent representation space semantic information preservation k-nearest neighbor relationships visual embeddings patch-level embedding reconstruction visually grounded question-answering tasks