Every Attention Matters: An Efficient Hybrid Architecture for
  Long-Context Reasoning
In this technical report, we present the Ring-linear model series,
specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0.
Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while
Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both
models adopt a hybrid architecture that effectively integrates linear attention
and softmax attention, significantly reducing I/O and computational overhead in
long-context inference scenarios. Compared to a 32 billion parameter dense
model, this series reduces inference cost to 1/10, and compared to the original
Ring series, the cost is also reduced by over 50%. Furthermore, through
systematic exploration of the ratio between different attention mechanisms in
the hybrid architecture, we have identified the currently optimal model
structure. Additionally, by leveraging our self-developed high-performance FP8
operator library-linghe, overall training efficiency has been improved by 50%.
Benefiting from the high alignment between the training and inference engine
operators, the models can undergo long-term, stable, and highly efficient
optimization during the reinforcement learning phase, consistently maintaining
SOTA performance across multiple challenging complex reasoning benchmarks.
68f98d4fb9b2e4ae046737dc
Ling Team
68f98d4fb9b2e4ae046737dd
Bin Han
68f98d4fb9b2e4ae046737de
Caizhi Tang
68f98d4fb9b2e4ae046737df
Chen Liang
68f98d4fb9b2e4ae046737e0
Donghao Zhang
68f98d4fb9b2e4ae046737e1
Fan Yuan
68f98d4fb9b2e4ae046737e2
Feng Zhu
68f98d4fb9b2e4ae046737e3
Jie Gao
68f98d4fb9b2e4ae046737e4
Jingyu Hu
claimed_verified
2025-10-23T14:49:07.705Z
64993d522ca6f96c8b8988ca
/avatars/0a8087cbbbcf5772288fe60c2c47eadb.svg
Jingyu, Hu
user
hjyai94
68f98d4fb9b2e4ae046737e5
Longfei Li
claimed_verified
2025-10-23T14:49:12.772Z
641c10cdbfe1bb9c4bce652f
/avatars/7ed954de541ac5bc80f58fb5db3866ea.svg
longfei li
user
long0x0
68f98d4fb9b2e4ae046737e6
Meng Li
68f98d4fb9b2e4ae046737e7
Mingyang Zhang
68f98d4fb9b2e4ae046737e8
Peijie Jiang
claimed_verified
2025-10-24T16:13:10.632Z
65afa634c1242ae261bc35d9
/avatars/d460aa6cc299a8080804e621875f314a.svg
Peijie Jiang
user
JZX555
68f98d4fb9b2e4ae046737e9
Peng Jiao
68f98d4fb9b2e4ae046737ea
Qian Zhao
claimed_verified
2025-10-23T14:48:55.104Z
5fde26773930f07f74aaf912
https://cdn-avatars.huggingface.co/v1/production/uploads/5fde26773930f07f74aaf912/SD93SVyVCRcTJbogBcf9J.jpeg
Qian Zhao
user
im0qianqian
68f98d4fb9b2e4ae046737eb
Qingyuan Yang
68f98d4fb9b2e4ae046737ec
Wenbo Shen
68f98d4fb9b2e4ae046737ed
Xinxing Yang
68f98d4fb9b2e4ae046737ee
Yalin Zhang
68f98d4fb9b2e4ae046737ef
Yankun Ren
68f98d4fb9b2e4ae046737f0
Yao Zhao
68f98d4fb9b2e4ae046737f1
Yibo Cao
68f98d4fb9b2e4ae046737f2
Yixuan Sun
68f98d4fb9b2e4ae046737f3
Yue Zhang
claimed_verified
2025-10-23T14:49:02.156Z
6430d8ad440a10209803d52f
https://cdn-avatars.huggingface.co/v1/production/uploads/6430d8ad440a10209803d52f/BDU_9ou3flPge_uwG_3b8.jpeg
Yue Zhang
user
York-Z
68f98d4fb9b2e4ae046737f4
Yuchen Fang
68f98d4fb9b2e4ae046737f5
Zibin Lin
68f98d4fb9b2e4ae046737f6
Zixuan Cheng
68f98d4fb9b2e4ae046737f7
Jun Zhou
Every Attention Matters: An Efficient Hybrid Architecture for
  Long-Context Reasoning
In this technical report, we present the Ring-linear model series,
specifically including Ring-mini-linear-2.0 and Ring-flash-linear-2.0.
Ring-mini-linear-2.0 comprises 16B parameters and 957M activations, while
Ring-flash-linear-2.0 contains 104B parameters and 6.1B activations. Both
models adopt a hybrid architecture that effectively integrates linear attention
and softmax attention, significantly reducing I/O and computational overhead in
long-context inference scenarios. Compared to a 32 billion parameter dense
model, this series reduces inference cost to 1/10, and compared to the original
Ring series, the cost is also reduced by over 50%. Furthermore, through
systematic exploration of the ratio between different attention mechanisms in
the hybrid architecture, we have identified the currently optimal model
structure. Additionally, by leveraging our self-developed high-performance FP8
operator library-linghe, overall training efficiency has been improved by 50%.
Benefiting from the high alignment between the training and inference engine
operators, the models can undergo long-term, stable, and highly efficient
optimization during the reinforcement learning phase, consistently maintaining
SOTA performance across multiple challenging complex reasoning benchmarks.
The Ring-linear model series, including Ring-mini-linear-2.0 and Ring-flash-linear-2.0, uses a hybrid architecture combining linear and softmax attention to reduce inference costs and improve training efficiency.
linear attention softmax attention hybrid architecture FP8 operator library reinforcement learning SOTA performance