Gated Associative Memory: A Parallel O(N) Architecture for Efficient
  Sequence Modeling
The Transformer architecture, underpinned by the self-attention mechanism,
has become the de facto standard for sequence modeling tasks. However, its core
computational primitive scales quadratically with sequence length (O(N^2)),
creating a significant bottleneck for processing long contexts. In this paper,
we propose the Gated Associative Memory (GAM) network, a novel, fully parallel
architecture for sequence modeling that exhibits linear complexity (O(N)) with
respect to sequence length. The GAM block replaces the self-attention layer
with two parallel pathways: a causal convolution to efficiently capture local,
position-dependent context, and a parallel associative memory retrieval
mechanism to model global, content-based patterns. These pathways are
dynamically fused using a gating mechanism, allowing the model to flexibly
combine local and global information for each token. We implement GAM from
scratch and conduct a rigorous comparative analysis against a standard
Transformer model and a modern linear-time baseline (Mamba) on the WikiText-2
benchmark, as well as against the Transformer on the TinyStories dataset. Our
experiments demonstrate that GAM is consistently faster, outperforming both
baselines on training speed, and achieves a superior or competitive final
validation perplexity across all datasets, establishing it as a promising and
efficient alternative for sequence modeling.
68b79ce2295f15ff60911296
Rishiraj Acharya
claimed_verified
2025-09-03T08:27:36.760Z
61030ed7d6edf00e0107a465
https://cdn-avatars.huggingface.co/v1/production/uploads/61030ed7d6edf00e0107a465/JP29xIxDfvQAsFgxpNkbp.png
Rishiraj Acharya
user
rishiraj
Gated Associative Memory: A Parallel O(N) Architecture for Efficient
  Sequence Modeling
The Transformer architecture, underpinned by the self-attention mechanism,
has become the de facto standard for sequence modeling tasks. However, its core
computational primitive scales quadratically with sequence length (O(N^2)),
creating a significant bottleneck for processing long contexts. In this paper,
we propose the Gated Associative Memory (GAM) network, a novel, fully parallel
architecture for sequence modeling that exhibits linear complexity (O(N)) with
respect to sequence length. The GAM block replaces the self-attention layer
with two parallel pathways: a causal convolution to efficiently capture local,
position-dependent context, and a parallel associative memory retrieval
mechanism to model global, content-based patterns. These pathways are
dynamically fused using a gating mechanism, allowing the model to flexibly
combine local and global information for each token. We implement GAM from
scratch and conduct a rigorous comparative analysis against a standard
Transformer model and a modern linear-time baseline (Mamba) on the WikiText-2
benchmark, as well as against the Transformer on the TinyStories dataset. Our
experiments demonstrate that GAM is consistently faster, outperforming both
baselines on training speed, and achieves a superior or competitive final
validation perplexity across all datasets, establishing it as a promising and
efficient alternative for sequence modeling.
Gated Associative Memory (GAM) network offers a linear complexity alternative to the Transformer architecture, improving training speed and achieving competitive validation perplexity.
Transformer architecture self-attention mechanism Gated Associative Memory (GAM) causal convolution associative memory retrieval gating mechanism WikiText-2 TinyStories validation perplexity