Memory Retrieval and Consolidation in Large Language Models through
  Function Tokens
The remarkable success of large language models (LLMs) stems from their
ability to consolidate vast amounts of knowledge into the memory during
pre-training and to retrieve it from the memory during inference, enabling
advanced capabilities such as knowledge memorization, instruction-following and
reasoning. However, the mechanisms of memory retrieval and consolidation in
LLMs remain poorly understood. In this paper, we propose the function token
hypothesis to explain the workings of LLMs: During inference, function tokens
activate the most predictive features from context and govern next token
prediction (memory retrieval). During pre-training, predicting the next tokens
(usually content tokens) that follow function tokens increases the number of
learned features of LLMs and updates the model parameters (memory
consolidation). Function tokens here roughly correspond to function words in
linguistics, including punctuation marks, articles, prepositions, and
conjunctions, in contrast to content tokens. We provide extensive experimental
evidence supporting this hypothesis. Using bipartite graph analysis, we show
that a small number of function tokens activate the majority of features. Case
studies further reveal how function tokens activate the most predictive
features from context to direct next token prediction. We also find that during
pre-training, the training loss is dominated by predicting the next content
tokens following function tokens, which forces the function tokens to select
the most predictive features from context.
68e8b3ee95e8e6771df38bd0
Shaohua Zhang
claimed_verified
2025-10-10T08:34:47.847Z
68e87503d08107e9fc8470ed
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/URRTpGTTswvOWSSu3QH0c.png
Shaohua Zhang
user
zhangshaohua
68e8b3ee95e8e6771df38bd1
Yuan Lin
68e8b3ee95e8e6771df38bd2
Hang Li
Memory Retrieval and Consolidation in Large Language Models through
  Function Tokens
The remarkable success of large language models (LLMs) stems from their
ability to consolidate vast amounts of knowledge into the memory during
pre-training and to retrieve it from the memory during inference, enabling
advanced capabilities such as knowledge memorization, instruction-following and
reasoning. However, the mechanisms of memory retrieval and consolidation in
LLMs remain poorly understood. In this paper, we propose the function token
hypothesis to explain the workings of LLMs: During inference, function tokens
activate the most predictive features from context and govern next token
prediction (memory retrieval). During pre-training, predicting the next tokens
(usually content tokens) that follow function tokens increases the number of
learned features of LLMs and updates the model parameters (memory
consolidation). Function tokens here roughly correspond to function words in
linguistics, including punctuation marks, articles, prepositions, and
conjunctions, in contrast to content tokens. We provide extensive experimental
evidence supporting this hypothesis. Using bipartite graph analysis, we show
that a small number of function tokens activate the majority of features. Case
studies further reveal how function tokens activate the most predictive
features from context to direct next token prediction. We also find that during
pre-training, the training loss is dominated by predicting the next content
tokens following function tokens, which forces the function tokens to select
the most predictive features from context.
Function tokens in large language models activate predictive features during inference and guide memory consolidation during pre-training by predicting subsequent content tokens.
large language models function tokens memory retrieval memory consolidation next token prediction bipartite graph analysis content tokens