Pretraining with hierarchical memories: separating long-tail and common
  knowledge
The impressive performance gains of modern language models currently rely on
scaling parameters: larger models store more world knowledge and reason better.
Yet compressing all world knowledge into parameters is unnecessary, as only a
fraction is used per prompt, and impractical for edge devices with limited
inference-time memory and compute. We address this shortcoming by a
memory-augmented architecture and a pretraining strategy aligned with existing
hardware paradigms. We introduce small language models that access large
hierarchical parametric memory banks encoding world knowledge. During
pretraining and inference, we fetch a small, context-dependent memory block and
add it to the model. Our pretraining learns to store long-tail world knowledge
in the memory parameters, while the small language model acts as an anchor
capturing common knowledge and general reasoning abilities. Through
trillion-token-scale experiments, we show significant gains: a 160M-parameters
model augmented with an 18M-parameters memory fetched from a 4.6B memory bank
obtains comparable performance to a regular model with more than 2x the
parameters. Through extensive experiments, we study the optimal type and size
of parametric memories in transformers, scaling them to over 21B parameters. We
find that our proposed hierarchical feed-forward memories work robustly across
transformer architectures, whether added during pretraining or post-hoc.
68e3cc4c73e20ab57784221e
Hadi Pouransari
68e3cc4c73e20ab57784221f
David Grangier
68e3cc4c73e20ab577842220
C Thomas
68e3cc4c73e20ab577842221
Michael Kirchhof
68e3cc4c73e20ab577842222
Oncel Tuzel
Pretraining with hierarchical memories: separating long-tail and common
  knowledge
The impressive performance gains of modern language models currently rely on
scaling parameters: larger models store more world knowledge and reason better.
Yet compressing all world knowledge into parameters is unnecessary, as only a
fraction is used per prompt, and impractical for edge devices with limited
inference-time memory and compute. We address this shortcoming by a
memory-augmented architecture and a pretraining strategy aligned with existing
hardware paradigms. We introduce small language models that access large
hierarchical parametric memory banks encoding world knowledge. During
pretraining and inference, we fetch a small, context-dependent memory block and
add it to the model. Our pretraining learns to store long-tail world knowledge
in the memory parameters, while the small language model acts as an anchor
capturing common knowledge and general reasoning abilities. Through
trillion-token-scale experiments, we show significant gains: a 160M-parameters
model augmented with an 18M-parameters memory fetched from a 4.6B memory bank
obtains comparable performance to a regular model with more than 2x the
parameters. Through extensive experiments, we study the optimal type and size
of parametric memories in transformers, scaling them to over 21B parameters. We
find that our proposed hierarchical feed-forward memories work robustly across
transformer architectures, whether added during pretraining or post-hoc.
A memory-augmented architecture with hierarchical parametric memory banks improves language model performance while reducing parameter size and computational requirements.
memory-augmented architecture parametric memory banks pretraining strategy hierarchical feed-forward memories transformers