How new data permeates LLM knowledge and how to dilute it
Large language models learn and continually learn through the accumulation of
gradient-based updates, but how individual pieces of new information affect
existing knowledge, leading to both beneficial generalization and problematic
hallucination, remains poorly understood. We demonstrate that when learning new
information, LLMs exhibit a "priming" effect: learning a new fact can cause the
model to inappropriately apply that knowledge in unrelated contexts. To
systematically study this phenomenon, we introduce "Outlandish," a carefully
curated dataset of 1320 diverse text samples designed to probe how new
knowledge permeates through an LLM's existing knowledge base. Using this
dataset, we show that the degree of priming after learning new information can
be predicted by measuring the token probability of key words before learning.
This relationship holds robustly across different model architectures (PALM-2,
Gemma, Llama), sizes, and training stages. Finally, we develop two novel
techniques to modulate how new knowledge affects existing model behavior: (1) a
``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update
pruning method. These approaches reduce undesirable priming effects by 50-95\%
while preserving the model's ability to learn new information. Our findings
provide both empirical insights into how LLMs learn and practical tools for
improving the specificity of knowledge insertion in language models. Further
materials: https://sunchipsster1.github.io/projects/outlandish/
67fe132f1d1bc292f7cdbc0f
Chen Sun
67fe132f1d1bc292f7cdbc10
Renat Aksitov
admin_assigned
2025-04-15T08:36:40.229Z
6003c87f532970af8c4d3a4a
/avatars/a4c5fbe427791d02e3cee208f22f18c4.svg
Renat Aksitov
user
mendor
67fe132f1d1bc292f7cdbc11
Andrey Zhmoginov
67fe132f1d1bc292f7cdbc12
Nolan Andrew Miller
67fe132f1d1bc292f7cdbc13
Max Vladymyrov
admin_assigned
2025-04-15T08:37:00.587Z
6502fe0bb1792803da806d42
/avatars/60280ce59f1f0d67e4210b0453039282.svg
Max Vladymyrov
user
gozzo87
67fe132f1d1bc292f7cdbc14
Ulrich Rueckert
67fe132f1d1bc292f7cdbc15
Been Kim
claimed_verified
2025-04-16T09:23:17.092Z
65dec807f4cf523519261714
/avatars/c6acb4f0c7ad715ff2c821c85be88d73.svg
been kim
user
Beenkim
67fe132f1d1bc292f7cdbc16
Mark Sandler
How new data permeates LLM knowledge and how to dilute it
Large language models learn and continually learn through the accumulation of
gradient-based updates, but how individual pieces of new information affect
existing knowledge, leading to both beneficial generalization and problematic
hallucination, remains poorly understood. We demonstrate that when learning new
information, LLMs exhibit a "priming" effect: learning a new fact can cause the
model to inappropriately apply that knowledge in unrelated contexts. To
systematically study this phenomenon, we introduce "Outlandish," a carefully
curated dataset of 1320 diverse text samples designed to probe how new
knowledge permeates through an LLM's existing knowledge base. Using this
dataset, we show that the degree of priming after learning new information can
be predicted by measuring the token probability of key words before learning.
This relationship holds robustly across different model architectures (PALM-2,
Gemma, Llama), sizes, and training stages. Finally, we develop two novel
techniques to modulate how new knowledge affects existing model behavior: (1) a
``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update
pruning method. These approaches reduce undesirable priming effects by 50-95\%
while preserving the model's ability to learn new information. Our findings
provide both empirical insights into how LLMs learn and practical tools for
improving the specificity of knowledge insertion in language models. Further
materials: https://sunchipsster1.github.io/projects/outlandish/
LLMs exhibit priming effects when learning new information, which can lead to inappropriate knowledge application; the degree of priming is predictable and modifiable through specific techniques.
gradient-based updates large language models priming effect Outlandish dataset token probability PALM-2 Gemma Llama stepping-stone text augmentation ignore-k update pruning