From Noise to Narrative: Tracing the Origins of Hallucinations in
  Transformers
As generative AI systems become competent and democratized in science,
business, and government, deeper insight into their failure modes now poses an
acute need. The occasional volatility in their behavior, such as the propensity
of transformer models to hallucinate, impedes trust and adoption of emerging AI
solutions in high-stakes areas. In the present work, we establish how and when
hallucinations arise in pre-trained transformer models through concept
representations captured by sparse autoencoders, under scenarios with
experimentally controlled uncertainty in the input space. Our systematic
experiments reveal that the number of semantic concepts used by the transformer
model grows as the input information becomes increasingly unstructured. In the
face of growing uncertainty in the input space, the transformer model becomes
prone to activate coherent yet input-insensitive semantic features, leading to
hallucinated output. At its extreme, for pure-noise inputs, we identify a wide
variety of robustly triggered and meaningful concepts in the intermediate
activations of pre-trained transformer models, whose functional integrity we
confirm through targeted steering. We also show that hallucinations in the
output of a transformer model can be reliably predicted from the concept
patterns embedded in transformer layer activations. This collection of insights
on transformer internal processing mechanics has immediate consequences for
aligning AI models with human values, AI safety, opening the attack surface for
potential adversarial attacks, and providing a basis for automatic
quantification of a model's hallucination risk.
68c1bc3a29b8ec9932cd080e
Praneet Suresh
claimed_verified
2025-09-12T16:16:15.804Z
62f678d2d278a8f3e7842621
/avatars/330cc1fb81f5c7c9f8276316c2ec186d.svg
Praneet Suresh
user
PraneetNeuro
68c1bc3a29b8ec9932cd080f
Jack Stanley
claimed_verified
2025-09-15T15:09:02.396Z
629266207bb0cb3c26321c63
/avatars/994bf572bff7f80976c6eaea4fa88b38.svg
Jack Stanley
user
jack-stanley
68c1bc3a29b8ec9932cd0810
Sonia Joseph
68c1bc3a29b8ec9932cd0811
Luca Scimeca
68c1bc3a29b8ec9932cd0812
Danilo Bzdok
From Noise to Narrative: Tracing the Origins of Hallucinations in
  Transformers
As generative AI systems become competent and democratized in science,
business, and government, deeper insight into their failure modes now poses an
acute need. The occasional volatility in their behavior, such as the propensity
of transformer models to hallucinate, impedes trust and adoption of emerging AI
solutions in high-stakes areas. In the present work, we establish how and when
hallucinations arise in pre-trained transformer models through concept
representations captured by sparse autoencoders, under scenarios with
experimentally controlled uncertainty in the input space. Our systematic
experiments reveal that the number of semantic concepts used by the transformer
model grows as the input information becomes increasingly unstructured. In the
face of growing uncertainty in the input space, the transformer model becomes
prone to activate coherent yet input-insensitive semantic features, leading to
hallucinated output. At its extreme, for pure-noise inputs, we identify a wide
variety of robustly triggered and meaningful concepts in the intermediate
activations of pre-trained transformer models, whose functional integrity we
confirm through targeted steering. We also show that hallucinations in the
output of a transformer model can be reliably predicted from the concept
patterns embedded in transformer layer activations. This collection of insights
on transformer internal processing mechanics has immediate consequences for
aligning AI models with human values, AI safety, opening the attack surface for
potential adversarial attacks, and providing a basis for automatic
quantification of a model's hallucination risk.
Transformer models tend to activate input-insensitive semantic features under uncertainty, leading to hallucinations that can be predicted from their internal activations.
transformer models hallucinations sparse autoencoders input space semantic concepts intermediate activations adversarial attacks hallucination risk