Large Language Models Do NOT Really Know What They Don't Know
Recent work suggests that large language models (LLMs) encode factuality
signals in their internal representations, such as hidden states, attention
weights, or token probabilities, implying that LLMs may "know what they don't
know". However, LLMs can also produce factual errors by relying on shortcuts or
spurious associations. These error are driven by the same training objective
that encourage correct predictions, raising the question of whether internal
computations can reliably distinguish between factual and hallucinated outputs.
In this work, we conduct a mechanistic analysis of how LLMs internally process
factual queries by comparing two types of hallucinations based on their
reliance on subject information. We find that when hallucinations are
associated with subject knowledge, LLMs employ the same internal recall process
as for correct responses, leading to overlapping and indistinguishable
hidden-state geometries. In contrast, hallucinations detached from subject
knowledge produce distinct, clustered representations that make them
detectable. These findings reveal a fundamental limitation: LLMs do not encode
truthfulness in their internal states but only patterns of knowledge recall,
demonstrating that "LLMs don't really know what they don't know".
68f1a6f96e0bef323a68fd64
Chi Seng Cheang
claimed_verified
2025-10-17T04:34:35.821Z
6454f1486f4ae9965626409c
/avatars/ac9d471b230f8babd03a1f35378d990e.svg
Chi Seng Cheang
user
chiseng-cheang
68f1a6f96e0bef323a68fd65
Hou Pong Chan
claimed_verified
2025-10-18T04:49:27.204Z
604f67ef0fe8ff3ec13d71ef
https://cdn-avatars.huggingface.co/v1/production/uploads/604f67ef0fe8ff3ec13d71ef/KhUwWvZ3OJ9nEee3B-SXO.png
Hou Pong (Ken) Chan
user
kenchan0226
68f1a6f96e0bef323a68fd66
Wenxuan Zhang
68f1a6f96e0bef323a68fd67
Yang Deng
claimed_verified
2025-10-17T05:56:48.685Z
68f1c7bc2faaae2d56c10bff
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xWST1Lh10BK7SRKBLjkFR.jpeg
Yang Deng
user
ydeng17
Large Language Models Do NOT Really Know What They Don't Know
Recent work suggests that large language models (LLMs) encode factuality
signals in their internal representations, such as hidden states, attention
weights, or token probabilities, implying that LLMs may "know what they don't
know". However, LLMs can also produce factual errors by relying on shortcuts or
spurious associations. These error are driven by the same training objective
that encourage correct predictions, raising the question of whether internal
computations can reliably distinguish between factual and hallucinated outputs.
In this work, we conduct a mechanistic analysis of how LLMs internally process
factual queries by comparing two types of hallucinations based on their
reliance on subject information. We find that when hallucinations are
associated with subject knowledge, LLMs employ the same internal recall process
as for correct responses, leading to overlapping and indistinguishable
hidden-state geometries. In contrast, hallucinations detached from subject
knowledge produce distinct, clustered representations that make them
detectable. These findings reveal a fundamental limitation: LLMs do not encode
truthfulness in their internal states but only patterns of knowledge recall,
demonstrating that "LLMs don't really know what they don't know".
LLMs process factual queries and hallucinations similarly when associated with subject knowledge, leading to indistinguishable internal representations, but produce distinct representations for hallucinations without subject knowledge.
large language models LLMs hidden states attention weights token probabilities factual queries hallucinations subject information internal recall process hidden-state geometries truthfulness knowledge recall