TruthPrInt: Mitigating LVLM Object Hallucination Via Latent
  Truthful-Guided Pre-Intervention
Object Hallucination (OH) has been acknowledged as one of the major
trustworthy challenges in Large Vision-Language Models (LVLMs). Recent
advancements in Large Language Models (LLMs) indicate that internal states,
such as hidden states, encode the "overall truthfulness" of generated
responses. However, it remains under-explored how internal states in LVLMs
function and whether they could serve as "per-token" hallucination indicators,
which is essential for mitigating OH. In this paper, we first conduct an
in-depth exploration of LVLM internal states in relation to OH issues and
discover that (1) LVLM internal states are high-specificity per-token
indicators of hallucination behaviors. Moreover, (2) different LVLMs encode
universal patterns of hallucinations in common latent subspaces, indicating
that there exist "generic truthful directions" shared by various LVLMs. Based
on these discoveries, we propose Truthful-Guided Pre-Intervention (TruthPrInt)
that first learns the truthful direction of LVLM decoding and then applies
truthful-guided inference-time intervention during LVLM decoding. We further
propose ComnHallu to enhance both cross-LVLM and cross-data hallucination
detection transferability by constructing and aligning hallucination latent
subspaces. We evaluate TruthPrInt in extensive experimental settings, including
in-domain and out-of-domain scenarios, over popular LVLMs and OH benchmarks.
Experimental results indicate that TruthPrInt significantly outperforms
state-of-the-art methods. Codes will be available at
https://github.com/jinhaoduan/TruthPrInt.
67d42fbcf00a232c91a7a6c2
Jinhao Duan
67d42fbcf00a232c91a7a6c3
Fei Kong
67d42fbcf00a232c91a7a6c4
Hao Cheng
67d42fbcf00a232c91a7a6c5
James Diffenderfer
67d42fbcf00a232c91a7a6c6
Bhavya Kailkhura
claimed_verified
2025-06-10T09:30:44.298Z
65cb79db6427380bc21261e2
/avatars/a003eb5d0955417329c1a4170ae65879.svg
Bhavya Kailkhura
user
bhavyakailkhura
67d42fbcf00a232c91a7a6c7
Lichao Sun
67d42fbcf00a232c91a7a6c8
Xiaofeng Zhu
67d42fbcf00a232c91a7a6c9
Xiaoshuang Shi
67d42fbcf00a232c91a7a6ca
Kaidi Xu
TruthPrInt: Mitigating LVLM Object Hallucination Via Latent
  Truthful-Guided Pre-Intervention
Object Hallucination (OH) has been acknowledged as one of the major
trustworthy challenges in Large Vision-Language Models (LVLMs). Recent
advancements in Large Language Models (LLMs) indicate that internal states,
such as hidden states, encode the "overall truthfulness" of generated
responses. However, it remains under-explored how internal states in LVLMs
function and whether they could serve as "per-token" hallucination indicators,
which is essential for mitigating OH. In this paper, we first conduct an
in-depth exploration of LVLM internal states in relation to OH issues and
discover that (1) LVLM internal states are high-specificity per-token
indicators of hallucination behaviors. Moreover, (2) different LVLMs encode
universal patterns of hallucinations in common latent subspaces, indicating
that there exist "generic truthful directions" shared by various LVLMs. Based
on these discoveries, we propose Truthful-Guided Pre-Intervention (TruthPrInt)
that first learns the truthful direction of LVLM decoding and then applies
truthful-guided inference-time intervention during LVLM decoding. We further
propose ComnHallu to enhance both cross-LVLM and cross-data hallucination
detection transferability by constructing and aligning hallucination latent
subspaces. We evaluate TruthPrInt in extensive experimental settings, including
in-domain and out-of-domain scenarios, over popular LVLMs and OH benchmarks.
Experimental results indicate that TruthPrInt significantly outperforms
state-of-the-art methods. Codes will be available at
https://github.com/jinhaoduan/TruthPrInt.
Internal states in Large Vision-Language Models serve as per-token indicators of hallucination, allowing for the development of methods to mitigate object hallucination.
Object Hallucination (OH) Large Vision-Language Models (LVLMs) hidden states internal states per-token hallucination indicators truthful direction Truthful-Guided Pre-Intervention (TruthPrInt) ComnHallu hallucination latent subspaces