MoM: Linear Sequence Modeling with Mixture-of-Memories
Linear sequence modeling methods, such as linear attention, state space
modeling, and linear RNNs, offer significant efficiency improvements by
reducing the complexity of training and inference. However, these methods
typically compress the entire input sequence into a single fixed-size memory
state, which leads to suboptimal performance on recall-intensive downstream
tasks. Drawing inspiration from neuroscience, particularly the brain's ability
to maintain robust long-term memory while mitigating "memory interference", we
introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes
multiple independent memory states, with a router network directing input
tokens to specific memory states. This approach greatly enhances the overall
memory capacity while minimizing memory interference. As a result, MoM performs
exceptionally well on recall-intensive tasks, surpassing existing linear
sequence modeling techniques. Despite incorporating multiple memory states, the
computation of each memory state remains linear in complexity, allowing MoM to
retain the linear-complexity advantage during training, while
constant-complexity during inference. Our experimental results show that MoM
significantly outperforms current linear sequence models on downstream language
tasks, particularly recall-intensive tasks, and even achieves performance
comparable to Transformer models. The code is released at
https://github.com/OpenSparseLLMs/MoM and is also released as a part of
https://github.com/OpenSparseLLMs/Linear-MoE.
67b6dc1ba7567156c6547880
Jusen Du
admin_assigned
2025-02-20T16:08:01.601Z
65003e857804f04a163328d9
/avatars/fe32150aabfde8d283b38ccebcf6982e.svg
Jusen Du
user
JusenK
67b6dc1ba7567156c6547881
Weigao Sun
extracted_confirmed
2025-03-04T08:10:49.466Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
67b6dc1ba7567156c6547882
Disen Lan
claimed_verified
2025-02-20T15:52:54.472Z
66ea643899af9ac3463639b1
/avatars/252d470e761a57834dee3dbc60dfefed.svg
Disen Lan
user
landisen
67b6dc1ba7567156c6547883
Jiaxi Hu
admin_assigned
2025-02-20T16:08:08.108Z
665dc35752ff9daa9ba5a4ed
/avatars/df8b01879d97e599b610fa51414d3a18.svg
Hu Jiaxi
user
Jiaxihu2
67b6dc1ba7567156c6547884
Yu Cheng
claimed_verified
2025-06-02T07:53:21.040Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
MoM: Linear Sequence Modeling with Mixture-of-Memories
Linear sequence modeling methods, such as linear attention, state space
modeling, and linear RNNs, offer significant efficiency improvements by
reducing the complexity of training and inference. However, these methods
typically compress the entire input sequence into a single fixed-size memory
state, which leads to suboptimal performance on recall-intensive downstream
tasks. Drawing inspiration from neuroscience, particularly the brain's ability
to maintain robust long-term memory while mitigating "memory interference", we
introduce a novel architecture called Mixture-of-Memories (MoM). MoM utilizes
multiple independent memory states, with a router network directing input
tokens to specific memory states. This approach greatly enhances the overall
memory capacity while minimizing memory interference. As a result, MoM performs
exceptionally well on recall-intensive tasks, surpassing existing linear
sequence modeling techniques. Despite incorporating multiple memory states, the
computation of each memory state remains linear in complexity, allowing MoM to
retain the linear-complexity advantage during training, while
constant-complexity during inference. Our experimental results show that MoM
significantly outperforms current linear sequence models on downstream language
tasks, particularly recall-intensive tasks, and even achieves performance
comparable to Transformer models. The code is released at
https://github.com/OpenSparseLLMs/MoM and is also released as a part of
https://github.com/OpenSparseLLMs/Linear-MoE.
Mixture-of-Memories (MoM) uses independent memory states to enhance recall in sequence modeling while maintaining computational efficiency, outperforming linear models and achieving performance comparable to Transformers.
linear attention state space modeling linear RNNs Mixture-of-Memories MoM memory states router network memory interference computational complexity linear-complexity constant-complexity downstream language tasks