TransMamba: Flexibly Switching between Transformer and Mamba
Transformers are the cornerstone of modern large language models, but their
quadratic computational complexity limits efficiency in long-sequence
processing. Recent advancements in Mamba, a state space model (SSM) with linear
complexity, offer promising efficiency gains but suffer from unstable
contextual learning and multitask generalization. This paper proposes
TransMamba, a novel framework that unifies Transformer and Mamba through shared
parameter matrices (e.g., QKV and CBx), and thus could dynamically switch
between attention and SSM mechanisms at different token lengths and layers. We
design the Memory converter to bridge Transformer and Mamba by converting
attention outputs into SSM-compatible states, ensuring seamless information
flow at TransPoints where the transformation happens. The TransPoint scheduling
is also thoroughly explored for further improvements. We conducted extensive
experiments demonstrating that TransMamba achieves superior training efficiency
and performance compared to baselines, and validated the deeper consistency
between Transformer and Mamba paradigms, offering a scalable solution for
next-generation sequence modeling.
67ecb89d0210bff02fd3591e
Yixing Li
extracted_pending
2025-04-02T04:10:06.783Z
66b4e3d850b87d84498bbc89
/avatars/526c0cabf3a2c019a13bb82fcc7b43e9.svg
YixingLi
user
Yixinglee
67ecb89d0210bff02fd3591f
Ruobing Xie
67ecb89d0210bff02fd35920
Zhen Yang
claimed_verified
2025-04-02T08:21:39.891Z
62c4057732fa66fedacca0db
/avatars/813e8c9fcd14dde1fdd415408f61bec2.svg
Zhen Yang
user
andyyang
67ecb89d0210bff02fd35921
Xingwu Sun
67ecb89d0210bff02fd35922
Shuaipeng Li
67ecb89d0210bff02fd35923
Weidong Han
claimed_verified
2025-04-13T19:27:30.584Z
615c2f8416284d83200e1105
/avatars/8e0c89925f714de58f0fc77265acd76f.svg
han weidong
user
dongdong2021
67ecb89d0210bff02fd35924
Zhanhui Kang
67ecb89d0210bff02fd35925
Yu Cheng
claimed_verified
2025-06-02T07:51:32.687Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
67ecb89d0210bff02fd35926
Chengzhong Xu
67ecb89d0210bff02fd35927
Di Wang
67ecb89d0210bff02fd35928
Jie Jiang
TransMamba: Flexibly Switching between Transformer and Mamba
Transformers are the cornerstone of modern large language models, but their
quadratic computational complexity limits efficiency in long-sequence
processing. Recent advancements in Mamba, a state space model (SSM) with linear
complexity, offer promising efficiency gains but suffer from unstable
contextual learning and multitask generalization. This paper proposes
TransMamba, a novel framework that unifies Transformer and Mamba through shared
parameter matrices (e.g., QKV and CBx), and thus could dynamically switch
between attention and SSM mechanisms at different token lengths and layers. We
design the Memory converter to bridge Transformer and Mamba by converting
attention outputs into SSM-compatible states, ensuring seamless information
flow at TransPoints where the transformation happens. The TransPoint scheduling
is also thoroughly explored for further improvements. We conducted extensive
experiments demonstrating that TransMamba achieves superior training efficiency
and performance compared to baselines, and validated the deeper consistency
between Transformer and Mamba paradigms, offering a scalable solution for
next-generation sequence modeling.
TransMamba combines Transformers and Mamba frameworks to leverage the strengths of both, achieving efficient and effective long-sequence processing through shared parameters and dynamic switching mechanisms.
Transformers Mamba state space model shared parameter matrices QKV CBx attention mechanisms Memory converter TransPoints TransPoint scheduling sequence modeling