Steering Knowledge Selection Behaviours in LLMs via SAE-Based
  Representation Engineering
Large language models (LLMs) can store a significant amount of factual
knowledge in their parameters. However, their parametric knowledge may conflict
with the information provided in the context -- this phenomenon, known as
context-memory knowledge conflicts, can lead to undesirable model
behaviour, such as reliance on outdated or incorrect information. Analysing the
internal activations of LLMs, we find that they can internally register the
signals of knowledge conflict at mid-layers. Such signals allow us to detect
whether a knowledge conflict occurs and use inference-time intervention
strategies to resolve it. In this work, we propose SpARE, a
training-free representation engineering method that uses pre-trained
sparse auto-encoders (SAEs) to control the knowledge selection behaviour of
LLMs. SpARE identifies the functional features that control the
knowledge selection behaviours and applies them to edit the internal
activations of LLMs at inference time. Our experimental results show that
SpARE can effectively control the usage of either knowledge source to
resolve knowledge conflict in open-domain question-answering tasks, surpassing
existing representation engineering methods (+10%) as well as contrastive
decoding methods (+15%).
671713db6914e88ea68a9ec6
Yu Zhao
claimed_verified
2024-10-23T07:34:48.086Z
654a500d9b8bd6406d431c0d
/avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
Yu Zhao
user
yuzhaouoe
671713db6914e88ea68a9ec7
Alessio Devoto
claimed_verified
2024-10-30T09:55:28.080Z
62180288d05d07ebf7de5f31
https://cdn-avatars.huggingface.co/v1/production/uploads/62180288d05d07ebf7de5f31/io2RsaRSjIDktQylXVvEB.jpeg
Alessio
user
alessiodevoto
671713db6914e88ea68a9ec8
Giwon Hong
671713db6914e88ea68a9ec9
Xiaotang Du
claimed_verified
2024-11-15T09:42:44.534Z
63d4698d1da8c28b7b8ab04a
/avatars/73d4f49018db03a59e72b69bb2d1dd0c.svg
Xiaotang Du
user
acDante
671713db6914e88ea68a9eca
Aryo Pradipta Gema
claimed_verified
2024-10-28T11:25:25.780Z
644f895e23d7eb05ca695054
/avatars/3fb04dd8544b403262bf98507de05453.svg
Aryo Pradipta Gema
user
aryopg
671713db6914e88ea68a9ecb
Hongru Wang
claimed_verified
2024-10-30T09:55:26.414Z
65f906e5c3dbdcae83ff7aac
https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
Hongru Wang
user
Merlin-Hongru
671713db6914e88ea68a9ecc
Kam-Fai Wong
671713db6914e88ea68a9ecd
Pasquale Minervini
claimed_verified
2024-10-30T09:55:24.894Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
Steering Knowledge Selection Behaviours in LLMs via SAE-Based
  Representation Engineering
Large language models (LLMs) can store a significant amount of factual
knowledge in their parameters. However, their parametric knowledge may conflict
with the information provided in the context -- this phenomenon, known as
context-memory knowledge conflicts, can lead to undesirable model
behaviour, such as reliance on outdated or incorrect information. Analysing the
internal activations of LLMs, we find that they can internally register the
signals of knowledge conflict at mid-layers. Such signals allow us to detect
whether a knowledge conflict occurs and use inference-time intervention
strategies to resolve it. In this work, we propose SpARE, a
training-free representation engineering method that uses pre-trained
sparse auto-encoders (SAEs) to control the knowledge selection behaviour of
LLMs. SpARE identifies the functional features that control the
knowledge selection behaviours and applies them to edit the internal
activations of LLMs at inference time. Our experimental results show that
SpARE can effectively control the usage of either knowledge source to
resolve knowledge conflict in open-domain question-answering tasks, surpassing
existing representation engineering methods (+10%) as well as contrastive
decoding methods (+15%).
A training-free method called SpARE uses pre-trained sparse auto-encoders to control knowledge selection in LLMs, effectively resolving context-memory knowledge conflicts in open-domain QA tasks.
LLMs parametric knowledge context-memory knowledge conflicts internal activations inference-time intervention SpARE sparse auto-encoders knowledge selection behaviour open-domain question-answering tasks contrastive decoding