Why Language Models Hallucinate
Like students facing hard exam questions, large language models sometimes
guess when uncertain, producing plausible yet incorrect statements instead of
admitting uncertainty. Such "hallucinations" persist even in state-of-the-art
systems and undermine trust. We argue that language models hallucinate because
the training and evaluation procedures reward guessing over acknowledging
uncertainty, and we analyze the statistical causes of hallucinations in the
modern training pipeline. Hallucinations need not be mysterious -- they
originate simply as errors in binary classification. If incorrect statements
cannot be distinguished from facts, then hallucinations in pretrained language
models will arise through natural statistical pressures. We then argue that
hallucinations persist due to the way most evaluations are graded -- language
models are optimized to be good test-takers, and guessing when uncertain
improves test performance. This "epidemic" of penalizing uncertain responses
can only be addressed through a socio-technical mitigation: modifying the
scoring of existing benchmarks that are misaligned but dominate leaderboards,
rather than introducing additional hallucination evaluations. This change may
steer the field toward more trustworthy AI systems.
68be5810c123124955ef60ac
Adam Tauman Kalai
68be5810c123124955ef60ad
Ofir Nachum
68be5810c123124955ef60ae
Santosh S. Vempala
68be5810c123124955ef60af
Edwin Zhang
Why Language Models Hallucinate
Like students facing hard exam questions, large language models sometimes
guess when uncertain, producing plausible yet incorrect statements instead of
admitting uncertainty. Such "hallucinations" persist even in state-of-the-art
systems and undermine trust. We argue that language models hallucinate because
the training and evaluation procedures reward guessing over acknowledging
uncertainty, and we analyze the statistical causes of hallucinations in the
modern training pipeline. Hallucinations need not be mysterious -- they
originate simply as errors in binary classification. If incorrect statements
cannot be distinguished from facts, then hallucinations in pretrained language
models will arise through natural statistical pressures. We then argue that
hallucinations persist due to the way most evaluations are graded -- language
models are optimized to be good test-takers, and guessing when uncertain
improves test performance. This "epidemic" of penalizing uncertain responses
can only be addressed through a socio-technical mitigation: modifying the
scoring of existing benchmarks that are misaligned but dominate leaderboards,
rather than introducing additional hallucination evaluations. This change may
steer the field toward more trustworthy AI systems.
Language models produce incorrect statements due to training and evaluation procedures that reward guessing over acknowledging uncertainty, leading to a need for socio-technical changes in benchmark scoring.
hallucinations binary classification uncertain responses trustworthy AI systems