Do I Know This Entity? Knowledge Awareness and Hallucinations in
  Language Models
Hallucinations in large language models are a widespread problem, yet the
mechanisms behind whether models will hallucinate are poorly understood,
limiting our ability to solve this problem. Using sparse autoencoders as an
interpretability tool, we discover that a key part of these mechanisms is
entity recognition, where the model detects if an entity is one it can recall
facts about. Sparse autoencoders uncover meaningful directions in the
representation space, these detect whether the model recognizes an entity, e.g.
detecting it doesn't know about an athlete or a movie. This suggests that
models can have self-knowledge: internal representations about their own
capabilities. These directions are causally relevant: capable of steering the
model to refuse to answer questions about known entities, or to hallucinate
attributes of unknown entities when it would otherwise refuse. We demonstrate
that despite the sparse autoencoders being trained on the base model, these
directions have a causal effect on the chat model's refusal behavior,
suggesting that chat finetuning has repurposed this existing mechanism.
Furthermore, we provide an initial exploration into the mechanistic role of
these directions in the model, finding that they disrupt the attention of
downstream heads that typically move entity attributes to the final token.
67404b754feb9ca7e53aae25
Javier Ferrando
admin_assigned
2024-11-22T13:19:27.390Z
61a5051ecd3eb3043f38e443
/avatars/bcc94a31fab7486ca9d018245a289fb0.svg
Javier Ferrando
user
javifer
67404b754feb9ca7e53aae26
Oscar Obeso
67404b754feb9ca7e53aae27
Senthooran Rajamanoharan
admin_assigned
2024-11-22T13:19:56.405Z
6477122c99a5ce743ccf2f55
/avatars/dee617374fc609e07eba3bcb2cd16810.svg
Senthooran Rajamanoharan
user
srdm
67404b754feb9ca7e53aae28
Neel Nanda
admin_assigned
2024-11-22T13:20:12.295Z
62669380c8bc5cf80ca97350
/avatars/6d5cd2261163308b82341c1ce28984d1.svg
Neel Nanda
user
NeelNanda
Do I Know This Entity? Knowledge Awareness and Hallucinations in
  Language Models
Hallucinations in large language models are a widespread problem, yet the
mechanisms behind whether models will hallucinate are poorly understood,
limiting our ability to solve this problem. Using sparse autoencoders as an
interpretability tool, we discover that a key part of these mechanisms is
entity recognition, where the model detects if an entity is one it can recall
facts about. Sparse autoencoders uncover meaningful directions in the
representation space, these detect whether the model recognizes an entity, e.g.
detecting it doesn't know about an athlete or a movie. This suggests that
models can have self-knowledge: internal representations about their own
capabilities. These directions are causally relevant: capable of steering the
model to refuse to answer questions about known entities, or to hallucinate
attributes of unknown entities when it would otherwise refuse. We demonstrate
that despite the sparse autoencoders being trained on the base model, these
directions have a causal effect on the chat model's refusal behavior,
suggesting that chat finetuning has repurposed this existing mechanism.
Furthermore, we provide an initial exploration into the mechanistic role of
these directions in the model, finding that they disrupt the attention of
downstream heads that typically move entity attributes to the final token.
Sparse autoencoders reveal that large language models can recognize known entities and have internal self-knowledge about their capabilities, influencing their tendency to hallucinate.
sparse autoencoders entity recognition representation space self-knowledge causal effect chat finetuning attention entity attributes