Native Hybrid Attention for Efficient Sequence Modeling
Transformers excel at sequence modeling but face quadratic complexity, while
linear attention offers improved efficiency but often compromises recall
accuracy over long contexts. In this work, we introduce Native Hybrid Attention
(NHA), a novel hybrid architecture of linear and full attention that integrates
both intra \& inter-layer hybridization into a unified layer design. NHA
maintains long-term context in key-value slots updated by a linear RNN, and
augments them with short-term tokens from a sliding window. A single
softmax attention operation is then applied over all keys and values,
enabling per-token and per-head context-dependent weighting without requiring
additional fusion parameters. The inter-layer behavior is controlled through a
single hyperparameter, the sliding window size, which allows smooth adjustment
between purely linear and full attention while keeping all layers structurally
uniform. Experimental results show that NHA surpasses Transformers and other
hybrid baselines on recall-intensive and commonsense reasoning tasks.
Furthermore, pretrained LLMs can be structurally hybridized with NHA, achieving
competitive accuracy while delivering significant efficiency gains. Code is
available at https://github.com/JusenD/NHA.
68e772e67ae125f9582e6bf3
Jusen Du
claimed_verified
2025-10-14T10:23:55.925Z
65003e857804f04a163328d9
/avatars/fe32150aabfde8d283b38ccebcf6982e.svg
Jusen Du
user
JusenK
68e772e67ae125f9582e6bf4
Jiaxi Hu
68e772e67ae125f9582e6bf5
Tao Zhang
68e772e67ae125f9582e6bf6
Weigao Sun
claimed_verified
2025-10-09T12:48:12.592Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
68e772e67ae125f9582e6bf7
Yu Cheng
Native Hybrid Attention for Efficient Sequence Modeling
Transformers excel at sequence modeling but face quadratic complexity, while
linear attention offers improved efficiency but often compromises recall
accuracy over long contexts. In this work, we introduce Native Hybrid Attention
(NHA), a novel hybrid architecture of linear and full attention that integrates
both intra \& inter-layer hybridization into a unified layer design. NHA
maintains long-term context in key-value slots updated by a linear RNN, and
augments them with short-term tokens from a sliding window. A single
softmax attention operation is then applied over all keys and values,
enabling per-token and per-head context-dependent weighting without requiring
additional fusion parameters. The inter-layer behavior is controlled through a
single hyperparameter, the sliding window size, which allows smooth adjustment
between purely linear and full attention while keeping all layers structurally
uniform. Experimental results show that NHA surpasses Transformers and other
hybrid baselines on recall-intensive and commonsense reasoning tasks.
Furthermore, pretrained LLMs can be structurally hybridized with NHA, achieving
competitive accuracy while delivering significant efficiency gains. Code is
available at https://github.com/JusenD/NHA.
Native Hybrid Attention (NHA) combines linear and full attention mechanisms to maintain long-term context while improving efficiency, outperforming Transformers in recall-intensive tasks and offering efficiency gains in pretrained LLMs.
Native Hybrid Attention NHA linear attention full attention intra-layer hybridization inter-layer hybridization key-value slots linear RNN sliding window softmax attention recall-intensive tasks commonsense reasoning tasks pretrained LLMs