UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior
  Long-Context Learning
While Mixture of Experts (MoE) models achieve remarkable efficiency by
activating only subsets of parameters, they suffer from high memory access
costs during inference. Memory-layer architectures offer an appealing
alternative with very few memory access, but previous attempts like UltraMem
have only matched the performance of 2-expert MoE models, falling significantly
short of state-of-the-art 8-expert configurations. We present UltraMemV2, a
redesigned memory-layer architecture that closes this performance gap. Our
approach introduces five key improvements: integrating memory layers into every
transformer block, simplifying value expansion with single linear projections,
adopting FFN-based value processing from PEER, implementing principled
parameter initialization, and rebalancing memory-to-FFN computation ratios.
Through extensive evaluation, we demonstrate that UltraMemV2 achieves
performance parity with 8-expert MoE models under same computation and
parameters but significantly low memory access. Notably, UltraMemV2 shows
superior performance on memory-intensive tasks, with improvements of +1.6
points on long-context memorization, +6.2 points on multi-round memorization,
and +7.9 points on in-context learning. We validate our approach at scale with
models up to 2.5B activated parameters from 120B total parameters, and
establish that activation density has greater impact on performance than total
sparse parameter count. Our work brings memory-layer architectures to
performance parity with state-of-the-art MoE models, presenting a compelling
alternative for efficient sparse computation.
68ae770e364411bea07df75d
Zihao Huang
claimed_verified
2025-08-27T07:10:44.445Z
65a62085576772f531e13856
/avatars/72c67a60422e333ea4e323f7480ae0b7.svg
Huang Zihao
user
FetchFortune
68ae770e364411bea07df75e
Yu Bao
68ae770e364411bea07df75f
Qiyang Min
claimed_verified
2025-09-05T14:17:02.376Z
645507b89d37c3fb33279fe3
/avatars/ec1122137c49204ab968182d1f726c35.svg
min
user
qiyang-attn
68ae770e364411bea07df760
Siyan Chen
68ae770e364411bea07df761
Ran Guo
68ae770e364411bea07df762
Hongzhi Huang
68ae770e364411bea07df763
Defa Zhu
68ae770e364411bea07df764
Yutao Zeng
claimed_verified
2025-08-27T15:00:16.834Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
68ae770e364411bea07df765
Banggu Wu
68ae770e364411bea07df766
Xun Zhou
68ae770e364411bea07df767
Siyuan Qiao
UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior
  Long-Context Learning
While Mixture of Experts (MoE) models achieve remarkable efficiency by
activating only subsets of parameters, they suffer from high memory access
costs during inference. Memory-layer architectures offer an appealing
alternative with very few memory access, but previous attempts like UltraMem
have only matched the performance of 2-expert MoE models, falling significantly
short of state-of-the-art 8-expert configurations. We present UltraMemV2, a
redesigned memory-layer architecture that closes this performance gap. Our
approach introduces five key improvements: integrating memory layers into every
transformer block, simplifying value expansion with single linear projections,
adopting FFN-based value processing from PEER, implementing principled
parameter initialization, and rebalancing memory-to-FFN computation ratios.
Through extensive evaluation, we demonstrate that UltraMemV2 achieves
performance parity with 8-expert MoE models under same computation and
parameters but significantly low memory access. Notably, UltraMemV2 shows
superior performance on memory-intensive tasks, with improvements of +1.6
points on long-context memorization, +6.2 points on multi-round memorization,
and +7.9 points on in-context learning. We validate our approach at scale with
models up to 2.5B activated parameters from 120B total parameters, and
establish that activation density has greater impact on performance than total
sparse parameter count. Our work brings memory-layer architectures to
performance parity with state-of-the-art MoE models, presenting a compelling
alternative for efficient sparse computation.
UltraMemV2, a redesigned memory-layer architecture, achieves performance parity with 8-expert MoE models while significantly reducing memory access costs.
Mixture of Experts MoE memory-layer architectures UltraMem UltraMemV2 transformer block value expansion FFN-based value processing PEER parameter initialization memory-to-FFN computation ratios long-context memorization multi-round memorization in-context learning activation density sparse computation