Analysing the Residual Stream of Language Models Under Knowledge
  Conflicts
Large language models (LLMs) can store a significant amount of factual
knowledge in their parameters. However, their parametric knowledge may conflict
with the information provided in the context. Such conflicts can lead to
undesirable model behaviour, such as reliance on outdated or incorrect
information. In this work, we investigate whether LLMs can identify knowledge
conflicts and whether it is possible to know which source of knowledge the
model will rely on by analysing the residual stream of the LLM. Through probing
tasks, we find that LLMs can internally register the signal of knowledge
conflict in the residual stream, which can be accurately detected by probing
the intermediate model activations. This allows us to detect conflicts within
the residual stream before generating the answers without modifying the input
or model parameters. Moreover, we find that the residual stream shows
significantly different patterns when the model relies on contextual knowledge
versus parametric knowledge to resolve conflicts. This pattern can be employed
to estimate the behaviour of LLMs when conflict happens and prevent unexpected
answers before producing the answers. Our analysis offers insights into how
LLMs internally manage knowledge conflicts and provides a foundation for
developing methods to control the knowledge selection processes.
6717ea90b954d4a1d5926d0e
Yu Zhao
claimed_verified
2024-10-23T07:33:06.204Z
654a500d9b8bd6406d431c0d
/avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
Yu Zhao
user
yuzhaouoe
6717ea90b954d4a1d5926d0f
Xiaotang Du
claimed_verified
2024-11-15T09:42:42.571Z
63d4698d1da8c28b7b8ab04a
/avatars/73d4f49018db03a59e72b69bb2d1dd0c.svg
Xiaotang Du
user
acDante
6717ea90b954d4a1d5926d10
Giwon Hong
6717ea90b954d4a1d5926d11
Aryo Pradipta Gema
claimed_verified
2024-10-28T11:25:18.371Z
644f895e23d7eb05ca695054
/avatars/3fb04dd8544b403262bf98507de05453.svg
Aryo Pradipta Gema
user
aryopg
6717ea90b954d4a1d5926d12
Alessio Devoto
claimed_verified
2024-10-30T09:55:16.409Z
62180288d05d07ebf7de5f31
https://cdn-avatars.huggingface.co/v1/production/uploads/62180288d05d07ebf7de5f31/io2RsaRSjIDktQylXVvEB.jpeg
Alessio
user
alessiodevoto
6717ea90b954d4a1d5926d13
Hongru Wang
claimed_verified
2024-10-30T09:55:14.807Z
65f906e5c3dbdcae83ff7aac
https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
Hongru Wang
user
Merlin-Hongru
6717ea90b954d4a1d5926d14
Xuanli He
6717ea90b954d4a1d5926d15
Kam-Fai Wong
6717ea90b954d4a1d5926d16
Pasquale Minervini
claimed_verified
2025-02-10T15:51:35.008Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
Analysing the Residual Stream of Language Models Under Knowledge
  Conflicts
Large language models (LLMs) can store a significant amount of factual
knowledge in their parameters. However, their parametric knowledge may conflict
with the information provided in the context. Such conflicts can lead to
undesirable model behaviour, such as reliance on outdated or incorrect
information. In this work, we investigate whether LLMs can identify knowledge
conflicts and whether it is possible to know which source of knowledge the
model will rely on by analysing the residual stream of the LLM. Through probing
tasks, we find that LLMs can internally register the signal of knowledge
conflict in the residual stream, which can be accurately detected by probing
the intermediate model activations. This allows us to detect conflicts within
the residual stream before generating the answers without modifying the input
or model parameters. Moreover, we find that the residual stream shows
significantly different patterns when the model relies on contextual knowledge
versus parametric knowledge to resolve conflicts. This pattern can be employed
to estimate the behaviour of LLMs when conflict happens and prevent unexpected
answers before producing the answers. Our analysis offers insights into how
LLMs internally manage knowledge conflicts and provides a foundation for
developing methods to control the knowledge selection processes.
LLMs can identify knowledge conflicts through their residual stream, allowing for detection and prediction of their reliance on contextual or parametric knowledge before generating responses.
LLMs knowledge conflicts residual stream intermediate model activations contextual knowledge parametric knowledge