Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
Large vision-language models (LVLMs) achieve strong performance on multimodal
tasks, yet they often default to their language prior (LP) -- memorized textual
patterns from pre-training while under-utilizing visual evidence. Prior
analyses of LP mostly rely on input-output probing, which fails to reveal the
internal mechanisms governing when and how vision influences model behavior. To
address this gap, we present the first systematic analysis of language prior
through the lens of chain-of-embedding, which examines the layer-wise
representation dynamics within LVLMs. Our analysis reveals a universal
phenomenon: each model exhibits a Visual Integration Point (VIP), a critical
layer at which visual information begins to meaningfully reshape hidden
representations and influence decoding. Building on this observation, we
introduce the Total Visual Integration (TVI) estimator, which aggregates
representation distance beyond the VIP to quantify how strongly visual query
influences response generation. Across 54 model-dataset combinations spanning 9
contemporary LVLMs and 6 benchmarks, we demonstrate that VIP consistently
emerges, and that TVI reliably predicts the strength of language prior. This
offers a principled toolkit for diagnosing and understanding language prior in
LVLMs.
68dc155f4159d1f2418f98d2
Lin Long
claimed_verified
2025-10-01T10:22:56.318Z
64be6ffa805e5b6457363108
https://cdn-avatars.huggingface.co/v1/production/uploads/64be6ffa805e5b6457363108/FH_gqlgyQvoF-QFE42Yjc.jpeg
Lin Long
user
Kylin-ll
68dc155f4159d1f2418f98d3
Changdae Oh
claimed_verified
2025-10-02T13:56:29.510Z
672fc8ede7c89e44c9757259
/avatars/caa0d0de519ea96992c81328c89b3843.svg
Changdae Oh
user
changdae
68dc155f4159d1f2418f98d4
Seongheon Park
68dc155f4159d1f2418f98d5
Yixuan Li
Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
Large vision-language models (LVLMs) achieve strong performance on multimodal
tasks, yet they often default to their language prior (LP) -- memorized textual
patterns from pre-training while under-utilizing visual evidence. Prior
analyses of LP mostly rely on input-output probing, which fails to reveal the
internal mechanisms governing when and how vision influences model behavior. To
address this gap, we present the first systematic analysis of language prior
through the lens of chain-of-embedding, which examines the layer-wise
representation dynamics within LVLMs. Our analysis reveals a universal
phenomenon: each model exhibits a Visual Integration Point (VIP), a critical
layer at which visual information begins to meaningfully reshape hidden
representations and influence decoding. Building on this observation, we
introduce the Total Visual Integration (TVI) estimator, which aggregates
representation distance beyond the VIP to quantify how strongly visual query
influences response generation. Across 54 model-dataset combinations spanning 9
contemporary LVLMs and 6 benchmarks, we demonstrate that VIP consistently
emerges, and that TVI reliably predicts the strength of language prior. This
offers a principled toolkit for diagnosing and understanding language prior in
LVLMs.
A systematic analysis of language prior in large vision-language models reveals a Visual Integration Point and introduces a Total Visual Integration estimator to quantify visual influence on response generation.
Large vision-language models language prior chain-of-embedding layer-wise representation dynamics Visual Integration Point Total Visual Integration estimator