Qwen2.5-1M Technical Report
We introduce Qwen2.5-1M, a series of models that extend the context length to
1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series
have significantly enhanced long-context capabilities through long-context
pre-training and post-training. Key techniques such as long data synthesis,
progressive pre-training, and multi-stage supervised fine-tuning are employed
to effectively enhance long-context performance while reducing training costs.
  To promote the use of long-context models among a broader user base, we
present and open-source our inference framework. This framework includes a
length extrapolation method that can expand the model context lengths by at
least four times, or even more, without additional training. To reduce
inference costs, we implement a sparse attention method along with chunked
prefill optimization for deployment scenarios and a sparsity refinement method
to improve precision. Additionally, we detail our optimizations in the
inference engine, including kernel optimization, pipeline parallelism, and
scheduling optimization, which significantly enhance overall inference
performance. By leveraging our inference framework, the Qwen2.5-1M models
achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million
tokens of context. This framework provides an efficient and powerful solution
for developing applications that require long-context processing using
open-source models.
  The Qwen2.5-1M series currently includes the open-source models
Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed
model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly
improved in long-context tasks without compromising performance in
short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model
significantly outperforms GPT-4o-mini in long-context tasks and supports
contexts eight times longer.
67986c83b5e71350993d28eb
An Yang
claimed_verified
2025-05-19T06:43:08.373Z
62088594a5943c8a8fc94560
https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
An Yang
user
yangapku
67986c83b5e71350993d28ec
Bowen Yu
admin_assigned
2025-01-28T13:56:10.598Z
6583ab7983a9e1460c67d876
/avatars/74400bc448c3f07e23a4cd53d68a6af7.svg
bowen
user
bowenYu
67986c83b5e71350993d28ed
Chengyuan Li
67986c83b5e71350993d28ee
Dayiheng Liu
admin_assigned
2025-01-28T13:56:44.491Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
67986c83b5e71350993d28ef
Fei Huang
67986c83b5e71350993d28f0
Haoyan Huang
67986c83b5e71350993d28f1
Jiandong Jiang
67986c83b5e71350993d28f2
Jianhong Tu
admin_assigned
2025-01-28T13:58:35.581Z
654bead777401b47e6424f88
/avatars/7bcbdbb051c93b004f0dc3ad36c4a0ce.svg
Jianhong Tu
user
ToviTu
67986c83b5e71350993d28f3
Jianwei Zhang
67986c83b5e71350993d28f4
Jingren Zhou
67986c83b5e71350993d28f5
Junyang Lin
admin_assigned
2025-01-28T13:57:31.261Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
67986c83b5e71350993d28f6
Kai Dang
67986c83b5e71350993d28f7
Kexin Yang
admin_assigned
2025-01-28T13:57:56.435Z
65b0b3957e5d5a4ecc750de0
/avatars/e0d79d3265ca4ad5c5411feb01043fb4.svg
Kexin Yang
user
dawn0929
67986c83b5e71350993d28f8
Le Yu
claimed_verified
2025-05-18T08:48:00.054Z
6455056fe4952d1c6cb45e49
/avatars/bbdcd4405497f11ff3774352277d1ec7.svg
Le Yu
user
vanillaOVO
67986c83b5e71350993d28f9
Mei Li
67986c83b5e71350993d28fa
Minmin Sun
admin_assigned
2025-01-28T13:58:27.795Z
636a390037d9329b4a007009
/avatars/a3c9117e104d4667e39e20ec83dc5cd6.svg
Minmin Sun
user
minminsun
67986c83b5e71350993d28fb
Qin Zhu
67986c83b5e71350993d28fc
Rui Men
67986c83b5e71350993d28fd
Tao He
67986c83b5e71350993d28fe
Weijia Xu
67986c83b5e71350993d28ff
Wenbiao Yin
67986c83b5e71350993d2900
Wenyuan Yu
admin_assigned
2025-01-28T13:58:11.021Z
63f4c99721eb234ab73dd112
/avatars/162e92d7aeb7de1c6ebf4d6e2bff33f5.svg
yu wenyuan
user
liuxinyijian
67986c83b5e71350993d2901
Xiafei Qiu
67986c83b5e71350993d2902
Xingzhang Ren
67986c83b5e71350993d2903
Xinlong Yang
claimed_verified
2025-02-04T14:48:49.659Z
6493eb93f453c3a7a2828362
/avatars/0a0e1f010465ab2042e532fc1f5b8053.svg
Yang Xinlong
user
Yangyy666
67986c83b5e71350993d2904
Yong Li
67986c83b5e71350993d2905
Zhiying Xu
67986c83b5e71350993d2906
Zipeng Zhang
Qwen2.5-1M Technical Report
We introduce Qwen2.5-1M, a series of models that extend the context length to
1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series
have significantly enhanced long-context capabilities through long-context
pre-training and post-training. Key techniques such as long data synthesis,
progressive pre-training, and multi-stage supervised fine-tuning are employed
to effectively enhance long-context performance while reducing training costs.
  To promote the use of long-context models among a broader user base, we
present and open-source our inference framework. This framework includes a
length extrapolation method that can expand the model context lengths by at
least four times, or even more, without additional training. To reduce
inference costs, we implement a sparse attention method along with chunked
prefill optimization for deployment scenarios and a sparsity refinement method
to improve precision. Additionally, we detail our optimizations in the
inference engine, including kernel optimization, pipeline parallelism, and
scheduling optimization, which significantly enhance overall inference
performance. By leveraging our inference framework, the Qwen2.5-1M models
achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million
tokens of context. This framework provides an efficient and powerful solution
for developing applications that require long-context processing using
open-source models.
  The Qwen2.5-1M series currently includes the open-source models
Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed
model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly
improved in long-context tasks without compromising performance in
short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model
significantly outperforms GPT-4o-mini in long-context tasks and supports
contexts eight times longer.
The Qwen2.5-1M series models extend context length to 1 million tokens with enhanced long-context capabilities, employing techniques like long data synthesis and progressive pre-training, and are supported by an open-source inference framework with sparse attention and kernel optimizations.
long-context pre-training long data synthesis progressive pre-training multi-stage supervised fine-tuning length extrapolation sparse attention chunked prefill optimization sparsity refinement kernel optimization pipeline parallelism scheduling optimization