Artificial Hippocampus Networks for Efficient Long-Context Modeling
Long-sequence modeling faces a fundamental trade-off between the efficiency
of compressive fixed-size memory in RNN-like models and the fidelity of
lossless growing memory in attention-based Transformers. Inspired by the
Multi-Store Model in cognitive science, we introduce a memory framework of
artificial neural networks. Our method maintains a sliding window of the
Transformer's KV cache as lossless short-term memory, while a learnable module
termed Artificial Hippocampus Network (AHN) recurrently compresses
out-of-window information into a fixed-size compact long-term memory. To
validate this framework, we instantiate AHNs using modern RNN-like
architectures, including Mamba2, DeltaNet, and Gated DeltaNet. Extensive
experiments on long-context benchmarks LV-Eval and InfiniteBench demonstrate
that AHN-augmented models consistently outperform sliding window baselines and
achieve performance comparable or even superior to full-attention models, while
substantially reducing computational and memory requirements. For instance,
augmenting the Qwen2.5-3B-Instruct with AHNs reduces inference FLOPs by 40.5%
and memory cache by 74.0%, while improving its average score on LV-Eval (128k
sequence length) from 4.41 to 5.88. Code is available at:
https://github.com/ByteDance-Seed/AHN.
68e713227ae125f9582e68fb
Yunhao Fang
68e713227ae125f9582e68fc
Weihao Yu
claimed_verified
2025-10-10T09:36:37.892Z
5df833bdda6d0311fd3d5403
https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png
Weihao Yu
user
whyu
68e713227ae125f9582e68fd
Shu Zhong
68e713227ae125f9582e68fe
Qinghao Ye
68e713227ae125f9582e68ff
Xuehan Xiong
68e713227ae125f9582e6900
Lai Wei
Artificial Hippocampus Networks for Efficient Long-Context Modeling
Long-sequence modeling faces a fundamental trade-off between the efficiency
of compressive fixed-size memory in RNN-like models and the fidelity of
lossless growing memory in attention-based Transformers. Inspired by the
Multi-Store Model in cognitive science, we introduce a memory framework of
artificial neural networks. Our method maintains a sliding window of the
Transformer's KV cache as lossless short-term memory, while a learnable module
termed Artificial Hippocampus Network (AHN) recurrently compresses
out-of-window information into a fixed-size compact long-term memory. To
validate this framework, we instantiate AHNs using modern RNN-like
architectures, including Mamba2, DeltaNet, and Gated DeltaNet. Extensive
experiments on long-context benchmarks LV-Eval and InfiniteBench demonstrate
that AHN-augmented models consistently outperform sliding window baselines and
achieve performance comparable or even superior to full-attention models, while
substantially reducing computational and memory requirements. For instance,
augmenting the Qwen2.5-3B-Instruct with AHNs reduces inference FLOPs by 40.5%
and memory cache by 74.0%, while improving its average score on LV-Eval (128k
sequence length) from 4.41 to 5.88. Code is available at:
https://github.com/ByteDance-Seed/AHN.
A memory framework combining short-term and long-term memory in neural networks improves long-sequence modeling efficiency and performance.
RNN-like models attention-based Transformers Multi-Store Model sliding window KV cache Artificial Hippocampus Network (AHN) Mamba2 DeltaNet Gated DeltaNet LV-Eval InfiniteBench inference FLOPs memory cache