InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long
  Adaptation
Long-sequence processing is a critical capability for modern large language
models. However, the self-attention mechanism in the standard Transformer
architecture faces severe computational and memory bottlenecks when processing
long sequences. While trainable sparse attention methods offer a promising
solution, existing approaches such as NSA introduce excessive extra parameters
and disrupt the conventional pretrain-on-short, finetune-on-long
workflow, resulting in slow convergence and difficulty in acceleration. To
overcome these limitations, we introduce dense-sparse switchable attention
framework, termed as InfLLM-V2. InfLLM-V2 is a trainable sparse attention that
seamlessly adapts models from short to long sequences. Specifically, InfLLM-V2
reuses dense attention parameters through parameter-free architecture
modification, maintaining consistency between short and long sequence
processing. Additionally, InfLLM-V2 ensures computational efficiency across all
sequence lengths, by using dense attention for short inputs and smoothly
transitioning to sparse attention for long sequences. To achieve practical
acceleration, we further introduce an efficient implementation of InfLLM-V2
that significantly reduces the computational overhead. Our experiments on
long-context understanding and chain-of-thought reasoning demonstrate that
InfLLM-V2 is 4times faster than dense attention while retaining 98.1% and
99.7% of the performance, respectively. Based on the InfLLM-V2 framework, we
have trained and open-sourced MiniCPM4.1
(https://huggingface.co/openbmb/MiniCPM4.1-8B), a hybrid reasoning model,
providing a reproducible implementation for the research community.
68db57fdd2bf1f4b15ec7593
Weilin Zhao
admin_assigned
2025-09-30T12:45:29.337Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
68db57fdd2bf1f4b15ec7594
Zihan Zhou
68db57fdd2bf1f4b15ec7595
Zhou Su
admin_assigned
2025-09-30T12:46:17.193Z
65e028993c9276bcc05233cd
/avatars/1bb4666670f3778d412c3b04b4f3e75a.svg
zhousu
user
zhousu
68db57fdd2bf1f4b15ec7596
Chaojun Xiao
admin_assigned
2025-09-30T12:36:30.210Z
608f6d72283d0a8d7be9d1f9
/avatars/7f499a37019359a3c488ba6cc11751fc.svg
Chaojun XIAO
user
xcjthu
68db57fdd2bf1f4b15ec7597
Yuxuan Li
68db57fdd2bf1f4b15ec7598
Yanghao Li
admin_assigned
2025-09-30T12:46:44.138Z
65dad3870af7e21ba473439f
/avatars/da542e7d68ae937bbdb791f17096bb1c.svg
Yanghao Li
user
FrozzZen
68db57fdd2bf1f4b15ec7599
Yudi Zhang
68db57fdd2bf1f4b15ec759a
Weilun Zhao
68db57fdd2bf1f4b15ec759b
Zhen Li
68db57fdd2bf1f4b15ec759c
Yuxiang Huang
68db57fdd2bf1f4b15ec759d
Ao Sun
68db57fdd2bf1f4b15ec759e
Xu Han
68db57fdd2bf1f4b15ec759f
Zhiyuan Liu
InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long
  Adaptation
Long-sequence processing is a critical capability for modern large language
models. However, the self-attention mechanism in the standard Transformer
architecture faces severe computational and memory bottlenecks when processing
long sequences. While trainable sparse attention methods offer a promising
solution, existing approaches such as NSA introduce excessive extra parameters
and disrupt the conventional pretrain-on-short, finetune-on-long
workflow, resulting in slow convergence and difficulty in acceleration. To
overcome these limitations, we introduce dense-sparse switchable attention
framework, termed as InfLLM-V2. InfLLM-V2 is a trainable sparse attention that
seamlessly adapts models from short to long sequences. Specifically, InfLLM-V2
reuses dense attention parameters through parameter-free architecture
modification, maintaining consistency between short and long sequence
processing. Additionally, InfLLM-V2 ensures computational efficiency across all
sequence lengths, by using dense attention for short inputs and smoothly
transitioning to sparse attention for long sequences. To achieve practical
acceleration, we further introduce an efficient implementation of InfLLM-V2
that significantly reduces the computational overhead. Our experiments on
long-context understanding and chain-of-thought reasoning demonstrate that
InfLLM-V2 is 4times faster than dense attention while retaining 98.1% and
99.7% of the performance, respectively. Based on the InfLLM-V2 framework, we
have trained and open-sourced MiniCPM4.1
(https://huggingface.co/openbmb/MiniCPM4.1-8B), a hybrid reasoning model,
providing a reproducible implementation for the research community.
A dense-sparse switchable attention framework, InfLLM-V2, enhances long-sequence processing in large language models by efficiently adapting between dense and sparse attention mechanisms.
self-attention mechanism Transformer architecture trainable sparse attention dense-sparse switchable attention InfLLM-V2 parameter-free architecture modification dense attention sparse attention long-context understanding chain-of-thought reasoning MiniCPM4.1