Confidence Regulation Neurons in Language Models
Despite their widespread use, the mechanisms by which large language models
(LLMs) represent and regulate uncertainty in next-token predictions remain
largely unexplored. This study investigates two critical components believed to
influence this uncertainty: the recently discovered entropy neurons and a new
set of components that we term token frequency neurons. Entropy neurons are
characterized by an unusually high weight norm and influence the final layer
normalization (LayerNorm) scale to effectively scale down the logits. Our work
shows that entropy neurons operate by writing onto an unembedding null space,
allowing them to impact the residual stream norm with minimal direct effect on
the logits themselves. We observe the presence of entropy neurons across a
range of models, up to 7 billion parameters. On the other hand, token frequency
neurons, which we discover and describe here for the first time, boost or
suppress each token's logit proportionally to its log frequency, thereby
shifting the output distribution towards or away from the unigram distribution.
Finally, we present a detailed case study where entropy neurons actively manage
confidence in the setting of induction, i.e. detecting and continuing repeated
subsequences.
667a74f59f501609d28e34b9
Alessandro Stolfo
admin_assigned
2024-06-25T13:38:41.156Z
64c7dda1d418013c77d3acec
/avatars/10bc3e8754e1a9815da57d3aec2f0bb3.svg
Alessandro Stolfo
user
alestolfo
667a74f59f501609d28e34ba
Ben Wu
667a74f59f501609d28e34bb
Wes Gurnee
admin_assigned
2024-06-25T13:39:02.040Z
631f446fdaa9591e52319035
/avatars/34d7ff8bc59aac0ed77a961596c4e4b1.svg
Wes Gurnee
user
wesg
667a74f59f501609d28e34bc
Yonatan Belinkov
admin_assigned
2024-06-25T13:39:07.898Z
614c57f1ee44bcfe57b366d6
/avatars/186a9aed84681246f48ed2a012c50def.svg
Yonatan Belinkov
user
belinkov
667a74f59f501609d28e34bd
Xingyi Song
admin_assigned
2024-06-25T13:39:17.590Z
62e795ac7c106973ac6515eb
/avatars/2eaa80217aae2a8dbd6cc350b7f35311.svg
Song
user
Xingyi
667a74f59f501609d28e34be
Mrinmaya Sachan
667a74f59f501609d28e34bf
Neel Nanda
admin_assigned
2024-06-25T13:39:26.400Z
62669380c8bc5cf80ca97350
/avatars/6d5cd2261163308b82341c1ce28984d1.svg
Neel Nanda
user
NeelNanda
Confidence Regulation Neurons in Language Models
Despite their widespread use, the mechanisms by which large language models
(LLMs) represent and regulate uncertainty in next-token predictions remain
largely unexplored. This study investigates two critical components believed to
influence this uncertainty: the recently discovered entropy neurons and a new
set of components that we term token frequency neurons. Entropy neurons are
characterized by an unusually high weight norm and influence the final layer
normalization (LayerNorm) scale to effectively scale down the logits. Our work
shows that entropy neurons operate by writing onto an unembedding null space,
allowing them to impact the residual stream norm with minimal direct effect on
the logits themselves. We observe the presence of entropy neurons across a
range of models, up to 7 billion parameters. On the other hand, token frequency
neurons, which we discover and describe here for the first time, boost or
suppress each token's logit proportionally to its log frequency, thereby
shifting the output distribution towards or away from the unigram distribution.
Finally, we present a detailed case study where entropy neurons actively manage
confidence in the setting of induction, i.e. detecting and continuing repeated
subsequences.
Entropy neurons and token frequency neurons in large language models influence uncertainty by scaling down logits and adjusting output distributions based on token frequency, respectively.
large language models entropy neurons token frequency neurons normalization logits unembedding null space residual stream norm unigram distribution induction