Understanding and Mitigating Bottlenecks of State Space Models through
  the Lens of Recency and Over-smoothing
Structured State Space Models (SSMs) have emerged as alternatives to
transformers. While SSMs are often regarded as effective in capturing
long-sequence dependencies, we rigorously demonstrate that they are inherently
limited by strong recency bias. Our empirical studies also reveal that this
bias impairs the models' ability to recall distant information and introduces
robustness issues. Our scaling experiments then discovered that deeper
structures in SSMs can facilitate the learning of long contexts. However,
subsequent theoretical analysis reveals that as SSMs increase in depth, they
exhibit another inevitable tendency toward over-smoothing, e.g., token
representations becoming increasingly indistinguishable. This fundamental
dilemma between recency and over-smoothing hinders the scalability of existing
SSMs. Inspired by our theoretical findings, we propose to polarize two channels
of the state transition matrices in SSMs, setting them to zero and one,
respectively, simultaneously addressing recency bias and over-smoothing.
Experiments demonstrate that our polarization technique consistently enhances
the associative recall accuracy of long-range tokens and unlocks SSMs to
benefit further from deeper architectures. All source codes are released at
https://github.com/VITA-Group/SSM-Bottleneck.
67777d99f215dad5eb2e85c6
Peihao Wang
67777d99f215dad5eb2e85c7
Ruisi Cai
67777d99f215dad5eb2e85c8
Yuehao Wang
67777d99f215dad5eb2e85c9
Jiajun Zhu
claimed_verified
2025-01-03T14:03:05.945Z
65dc42ad1ed8dc6d82e3e4d8
/avatars/d2001d58b4fff24509c9e8e952042fe5.svg
Jiajun Zhu
user
lanczos
67777d99f215dad5eb2e85ca
Pragya Srivastava
extracted_confirmed
2025-01-03T06:05:01.330Z
639ccab166106be1436e1640
/avatars/1e3806e18ac427be20e93e5400f153d4.svg
Pragya Srivastava
user
pragsri8
67777d99f215dad5eb2e85cb
Zhangyang Wang
67777d99f215dad5eb2e85cc
Pan Li
extracted_pending
2025-07-28T23:05:37.001Z
68880233838fbe4aeb4f4ae6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nb6VBw_nrLNvlG8rMQ14F.png
Pan Li
user
lipan00123
Understanding and Mitigating Bottlenecks of State Space Models through
  the Lens of Recency and Over-smoothing
Structured State Space Models (SSMs) have emerged as alternatives to
transformers. While SSMs are often regarded as effective in capturing
long-sequence dependencies, we rigorously demonstrate that they are inherently
limited by strong recency bias. Our empirical studies also reveal that this
bias impairs the models' ability to recall distant information and introduces
robustness issues. Our scaling experiments then discovered that deeper
structures in SSMs can facilitate the learning of long contexts. However,
subsequent theoretical analysis reveals that as SSMs increase in depth, they
exhibit another inevitable tendency toward over-smoothing, e.g., token
representations becoming increasingly indistinguishable. This fundamental
dilemma between recency and over-smoothing hinders the scalability of existing
SSMs. Inspired by our theoretical findings, we propose to polarize two channels
of the state transition matrices in SSMs, setting them to zero and one,
respectively, simultaneously addressing recency bias and over-smoothing.
Experiments demonstrate that our polarization technique consistently enhances
the associative recall accuracy of long-range tokens and unlocks SSMs to
benefit further from deeper architectures. All source codes are released at
https://github.com/VITA-Group/SSM-Bottleneck.
Polarizing state transition matrices in Structured State Space Models addresses recency bias and over-smoothing, improving long-range token recall and enabling deeper architectures.
Structured State Space Models SSMs recency bias state transition matrices long-context learning over-smoothing associative recall accuracy token representations deep architectures