On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in
  Large Vision-Language Models
Large vision-language models (LVLMs), which integrate a vision encoder (VE)
with a large language model, have achieved remarkable success across various
tasks. However, there are still crucial challenges in LVLMs such as object
hallucination, generating descriptions of objects that are not in the input
image. Here, we argue that uncertain visual tokens within the VE is a key
factor that contributes to object hallucination. Our statistical analysis found
that there are positive correlations between visual tokens with high epistemic
uncertainty and the occurrence of hallucinations. Furthermore, we show
theoretically and empirically that visual tokens in early VE layers that
exhibit large representation deviations under small adversarial perturbations
indicate high epistemic uncertainty. Based on these findings, we propose a
simple yet effective strategy to mitigate object hallucination by modifying the
VE only. Our method comprises a proxy method with adversarial perturbations for
identifying uncertain visual tokens efficiently and a method to mask these
uncertain visual tokens during the self-attention process in the middle layers
of the VE, suppressing their influence on visual encoding and thus alleviating
hallucinations. Extensive experiments show that our method significantly
reduces object hallucinations in LVLMs and can synergistically work with other
prior arts.
68edc6a9de1fee572713a8fe
Hoigi Seo
claimed_verified
2025-10-14T07:30:10.592Z
633e6f07309a99325095dd42
/avatars/57b91a488ac1745b3c0509c04eb6ad93.svg
Hoigi Seo
user
Agorium
68edc6a9de1fee572713a8ff
Dong Un Kang
68edc6a9de1fee572713a900
Hyunjin Cho
claimed_verified
2025-10-14T07:30:08.146Z
66b46b576badc45885923979
/avatars/622e95c50876d48fc2dcb9b0dbc74607.svg
Hyunjin Cho
user
jfdkjjs
68edc6a9de1fee572713a901
Joohoon Lee
68edc6a9de1fee572713a902
Se Young Chun
On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in
  Large Vision-Language Models
Large vision-language models (LVLMs), which integrate a vision encoder (VE)
with a large language model, have achieved remarkable success across various
tasks. However, there are still crucial challenges in LVLMs such as object
hallucination, generating descriptions of objects that are not in the input
image. Here, we argue that uncertain visual tokens within the VE is a key
factor that contributes to object hallucination. Our statistical analysis found
that there are positive correlations between visual tokens with high epistemic
uncertainty and the occurrence of hallucinations. Furthermore, we show
theoretically and empirically that visual tokens in early VE layers that
exhibit large representation deviations under small adversarial perturbations
indicate high epistemic uncertainty. Based on these findings, we propose a
simple yet effective strategy to mitigate object hallucination by modifying the
VE only. Our method comprises a proxy method with adversarial perturbations for
identifying uncertain visual tokens efficiently and a method to mask these
uncertain visual tokens during the self-attention process in the middle layers
of the VE, suppressing their influence on visual encoding and thus alleviating
hallucinations. Extensive experiments show that our method significantly
reduces object hallucinations in LVLMs and can synergistically work with other
prior arts.
A method to reduce object hallucinations in large vision-language models by identifying and masking uncertain visual tokens in the vision encoder.
vision-language models vision encoder object hallucination visual tokens epistemic uncertainty adversarial perturbations self-attention visual encoding