A Systematic Analysis of Hybrid Linear Attention
Transformers face quadratic complexity and memory issues with long sequences,
prompting the adoption of linear attention mechanisms using fixed-size hidden
states. However, linear models often suffer from limited recall performance,
leading to hybrid architectures that combine linear and full attention layers.
Despite extensive hybrid architecture research, the choice of linear attention
component has not been deeply explored. We systematically evaluate various
linear attention models across generations - vector recurrences to advanced
gating mechanisms - both standalone and hybridized. To enable this
comprehensive analysis, we trained and open-sourced 72 models: 36 at 340M
parameters (20B tokens) and 36 at 1.3B parameters (100B tokens), covering six
linear attention variants across five hybridization ratios. Benchmarking on
standard language modeling and recall tasks reveals that superior standalone
linear models do not necessarily excel in hybrids. While language modeling
remains stable across linear-to-full attention ratios, recall significantly
improves with increased full attention layers, particularly below a 3:1 ratio.
Our study highlights selective gating, hierarchical recurrence, and controlled
forgetting as critical for effective hybrid models. We recommend architectures
such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1
to achieve Transformer-level recall efficiently. Our models are open-sourced at
https://huggingface.co/collections/m-a-p/hybrid-linear-attention-research-686c488a63d609d2f20e2b1e.
686f2371d938c25d68441b36
Dustin Wang
686f2371d938c25d68441b37
Rui-Jie Zhu
claimed_verified
2025-07-10T07:09:30.737Z
63ff09f24852102d4871c19c
https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
Rui-Jie Zhu
user
ridger
686f2371d938c25d68441b38
Steven Abreu
686f2371d938c25d68441b39
Yong Shan
claimed_verified
2025-07-15T19:11:16.707Z
64b8cc7ebe76d2ff0703bfb3
/avatars/f1c7ff17fd923f1460d362333d9fbfe3.svg
yong
user
yo37
686f2371d938c25d68441b3a
Taylor Kergan
686f2371d938c25d68441b3b
Yuqi Pan
686f2371d938c25d68441b3c
Yuhong Chou
686f2371d938c25d68441b3d
Zheng Li
686f2371d938c25d68441b3e
Ge Zhang
claimed_verified
2025-07-22T14:00:57.520Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
686f2371d938c25d68441b3f
Wenhao Huang
686f2371d938c25d68441b40
Jason Eshraghian
A Systematic Analysis of Hybrid Linear Attention
Transformers face quadratic complexity and memory issues with long sequences,
prompting the adoption of linear attention mechanisms using fixed-size hidden
states. However, linear models often suffer from limited recall performance,
leading to hybrid architectures that combine linear and full attention layers.
Despite extensive hybrid architecture research, the choice of linear attention
component has not been deeply explored. We systematically evaluate various
linear attention models across generations - vector recurrences to advanced
gating mechanisms - both standalone and hybridized. To enable this
comprehensive analysis, we trained and open-sourced 72 models: 36 at 340M
parameters (20B tokens) and 36 at 1.3B parameters (100B tokens), covering six
linear attention variants across five hybridization ratios. Benchmarking on
standard language modeling and recall tasks reveals that superior standalone
linear models do not necessarily excel in hybrids. While language modeling
remains stable across linear-to-full attention ratios, recall significantly
improves with increased full attention layers, particularly below a 3:1 ratio.
Our study highlights selective gating, hierarchical recurrence, and controlled
forgetting as critical for effective hybrid models. We recommend architectures
such as HGRN-2 or GatedDeltaNet with a linear-to-full ratio between 3:1 and 6:1
to achieve Transformer-level recall efficiently. Our models are open-sourced at
https://huggingface.co/collections/m-a-p/hybrid-linear-attention-research-686c488a63d609d2f20e2b1e.
Research evaluates various linear attention models in hybrid architectures, finding that selective gating, hierarchical recurrence, and controlled forgetting are crucial for effective recall in Transformers.
transformers quadratic complexity memory issues long sequences linear attention mechanisms fixed-size hidden states recall performance hybrid architectures vector recurrences gating mechanisms HGRN-2 GatedDeltaNet