Understanding and Mitigating Bottlenecks of State Space Models through
  the Lens of Recency and Over-smoothing
Structured State Space Models (SSMs) have emerged as alternatives to
transformers. While SSMs are often regarded as effective in capturing
long-sequence dependencies, we rigorously demonstrate that they are inherently
limited by strong recency bias. Our empirical studies also reveal that this
bias impairs the models' ability to recall distant information and introduces
robustness issues. Our scaling experiments then discovered that deeper
structures in SSMs can facilitate the learning of long contexts. However,
subsequent theoretical analysis reveals that as SSMs increase in depth, they
exhibit another inevitable tendency toward over-smoothing, e.g., token
representations becoming increasingly indistinguishable. This fundamental
dilemma between recency and over-smoothing hinders the scalability of existing
SSMs. Inspired by our theoretical findings, we propose to polarize two channels
of the state transition matrices in SSMs, setting them to zero and one,
respectively, simultaneously addressing recency bias and over-smoothing.
Experiments demonstrate that our polarization technique consistently enhances
the associative recall accuracy of long-range tokens and unlocks SSMs to
benefit further from deeper architectures. All source codes are released at
https://github.com/VITA-Group/SSM-Bottleneck.
67777d99f215dad5eb2e85c6
Peihao Wang
67777d99f215dad5eb2e85c7
Ruisi Cai
67777d99f215dad5eb2e85c8
Yuehao Wang
67777d99f215dad5eb2e85c9
Jiajun Zhu
claimed_verified
2025-01-03T14:03:05.945Z
65dc42ad1ed8dc6d82e3e4d8
/avatars/d2001d58b4fff24509c9e8e952042fe5.svg
Jiajun Zhu
user
lanczos
67777d99f215dad5eb2e85ca
Pragya Srivastava
extracted_confirmed
2025-01-03T06:05:01.330Z
639ccab166106be1436e1640
/avatars/1e3806e18ac427be20e93e5400f153d4.svg
Pragya Srivastava
user
pragsri8
67777d99f215dad5eb2e85cb
Zhangyang Wang
67777d99f215dad5eb2e85cc
Pan Li
Understanding and Mitigating Bottlenecks of State Space Models through
  the Lens of Recency and Over-smoothing
Structured State Space Models (SSMs) have emerged as alternatives to
transformers. While SSMs are often regarded as effective in capturing
long-sequence dependencies, we rigorously demonstrate that they are inherently
limited by strong recency bias. Our empirical studies also reveal that this
bias impairs the models' ability to recall distant information and introduces
robustness issues. Our scaling experiments then discovered that deeper
structures in SSMs can facilitate the learning of long contexts. However,
subsequent theoretical analysis reveals that as SSMs increase in depth, they
exhibit another inevitable tendency toward over-smoothing, e.g., token
representations becoming increasingly indistinguishable. This fundamental
dilemma between recency and over-smoothing hinders the scalability of existing
SSMs. Inspired by our theoretical findings, we propose to polarize two channels
of the state transition matrices in SSMs, setting them to zero and one,
respectively, simultaneously addressing recency bias and over-smoothing.
Experiments demonstrate that our polarization technique consistently enhances
the associative recall accuracy of long-range tokens and unlocks SSMs to
benefit further from deeper architectures. All source codes are released at
https://github.com/VITA-Group/SSM-Bottleneck.
Rapid State Space Models (SSMs) long-sequence dependencies recency bias distant information robustness issues over-smoothing token representations state transition matrices polarization technique associative recall accuracy long-range tokens