Truth Neurons
Despite their remarkable success and deployment across diverse workflows,
language models sometimes produce untruthful responses. Our limited
understanding of how truthfulness is mechanistically encoded within these
models jeopardizes their reliability and safety. In this paper, we propose a
method for identifying representations of truthfulness at the neuron level. We
show that language models contain truth neurons, which encode truthfulness in a
subject-agnostic manner. Experiments conducted across models of varying scales
validate the existence of truth neurons, confirming that the encoding of
truthfulness at the neuron level is a property shared by many language models.
The distribution patterns of truth neurons over layers align with prior
findings on the geometry of truthfulness. Selectively suppressing the
activations of truth neurons found through the TruthfulQA dataset degrades
performance both on TruthfulQA and on other benchmarks, showing that the
truthfulness mechanisms are not tied to a specific dataset. Our results offer
novel insights into the mechanisms underlying truthfulness in language models
and highlight potential directions toward improving their trustworthiness and
reliability.
682d31dd17608739046e1169
Haohang Li
admin_assigned
2025-05-21T10:29:09.565Z
634cabd104491d9f7111eea3
https://cdn-avatars.huggingface.co/v1/production/uploads/634cabd104491d9f7111eea3/JoqlugwfD1aGkd-wZTmP7.jpeg
Haohang Li
user
Acatsama
682d31dd17608739046e116a
Yupeng Cao
admin_assigned
2025-05-21T10:29:23.338Z
62dd8f328456396d4f8aa894
https://cdn-avatars.huggingface.co/v1/production/uploads/62dd8f328456396d4f8aa894/mxxTBDR914By3f4ICEm7S.png
Yupeng Cao
user
YupengCao
682d31dd17608739046e116b
Yangyang Yu
admin_assigned
2025-05-21T10:29:31.615Z
64f757c6016d60f3199ef5e6
/avatars/2659ba698081265d0480b08161718013.svg
Yangyang Yu
user
ShirleyY
682d31dd17608739046e116c
Jordan W. Suchow
admin_assigned
2025-05-21T10:29:41.392Z
631a8c9e0867652f538a64e5
/avatars/09e1cef9e7d6f7c51579448d4605850e.svg
Jordan Suchow
user
jordansuchow
682d31dd17608739046e116d
Zining Zhu
admin_assigned
2025-05-21T10:29:50.430Z
62d63a9dc5ada8ef841b4787
/avatars/a79a4ac07984d9a8623c99bdce9add54.svg
Zining Zhu
user
ZiningZhu
Truth Neurons
Despite their remarkable success and deployment across diverse workflows,
language models sometimes produce untruthful responses. Our limited
understanding of how truthfulness is mechanistically encoded within these
models jeopardizes their reliability and safety. In this paper, we propose a
method for identifying representations of truthfulness at the neuron level. We
show that language models contain truth neurons, which encode truthfulness in a
subject-agnostic manner. Experiments conducted across models of varying scales
validate the existence of truth neurons, confirming that the encoding of
truthfulness at the neuron level is a property shared by many language models.
The distribution patterns of truth neurons over layers align with prior
findings on the geometry of truthfulness. Selectively suppressing the
activations of truth neurons found through the TruthfulQA dataset degrades
performance both on TruthfulQA and on other benchmarks, showing that the
truthfulness mechanisms are not tied to a specific dataset. Our results offer
novel insights into the mechanisms underlying truthfulness in language models
and highlight potential directions toward improving their trustworthiness and
reliability.
Truth neurons encode truthfulness in language models at a subject-agnostic level, and suppressing these neurons degrades performance across benchmarks.
truth neurons language models truthfulness neuron level geometry of truthfulness TruthfulQA dataset