QwenLong-CPRS: Towards infty-LLMs with Dynamic Context Optimization
This technical report presents QwenLong-CPRS, a context compression framework
designed for explicit long-context optimization, addressing prohibitive
computation overhead during the prefill stage and the "lost in the middle"
performance degradation of large language models (LLMs) during long sequence
processing. Implemented through a novel dynamic context optimization mechanism,
QwenLong-CPRS enables multi-granularity context compression guided by natural
language instructions, achieving both efficiency gains and improved
performance.
  Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key
innovations: (1) Natural language-guided dynamic optimization, (2)
Bidirectional reasoning layers for enhanced boundary awareness, (3) Token
critic mechanisms with language modeling heads, and (4) Window-parallel
inference.
  Comprehensive evaluations across five benchmarks (4K-2M word contexts)
demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority
over other context management methods like RAG and sparse attention in both
accuracy and efficiency. (2) Architecture-agnostic integration with all
flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3,
and Qwen2.5-max, achieves 21.59times context compression alongside
19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct,
QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on
Ruler-128K and InfiniteBench, establishing new SOTA performance.
6833ea049f968fc5c6b64486
Weizhou Shen
6833ea049f968fc5c6b64487
Chenliang Li
6833ea049f968fc5c6b64488
Fanqi Wan
claimed_verified
2025-05-26T08:09:25.991Z
62ecbffd99112e99c5f7fded
https://cdn-avatars.huggingface.co/v1/production/uploads/62ecbffd99112e99c5f7fded/U6iXAJbpm2vaC5qksEPiH.png
Fanqi Wan
user
Wanfq
6833ea049f968fc5c6b64489
Shengyi Liao
claimed_verified
2025-05-27T12:43:57.415Z
64a69c1a9704e9888c1da827
/avatars/096aefab0ca99af76097cdfc5e752402.svg
LiaoShengyi
user
LsyLsyLsyyy
6833ea049f968fc5c6b6448a
Shaopeng Lai
6833ea049f968fc5c6b6448b
Bo Zhang
6833ea049f968fc5c6b6448c
Yingcheng Shi
claimed_verified
2025-05-27T07:54:57.679Z
64a3897a34612d376415545c
/avatars/3d69ce7f59783e51fd8a97830333ead6.svg
shiyingcheng
user
shiyingcheng
6833ea049f968fc5c6b6448d
Yuning Wu
6833ea049f968fc5c6b6448e
Gang Fu
6833ea049f968fc5c6b6448f
Zhansheng Li
6833ea049f968fc5c6b64490
Bin Yang
6833ea049f968fc5c6b64491
Ji Zhang
6833ea049f968fc5c6b64492
Fei Huang
6833ea049f968fc5c6b64493
Jingren Zhou
6833ea049f968fc5c6b64494
Ming Yan
QwenLong-CPRS: Towards infty-LLMs with Dynamic Context Optimization
This technical report presents QwenLong-CPRS, a context compression framework
designed for explicit long-context optimization, addressing prohibitive
computation overhead during the prefill stage and the "lost in the middle"
performance degradation of large language models (LLMs) during long sequence
processing. Implemented through a novel dynamic context optimization mechanism,
QwenLong-CPRS enables multi-granularity context compression guided by natural
language instructions, achieving both efficiency gains and improved
performance.
  Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key
innovations: (1) Natural language-guided dynamic optimization, (2)
Bidirectional reasoning layers for enhanced boundary awareness, (3) Token
critic mechanisms with language modeling heads, and (4) Window-parallel
inference.
  Comprehensive evaluations across five benchmarks (4K-2M word contexts)
demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority
over other context management methods like RAG and sparse attention in both
accuracy and efficiency. (2) Architecture-agnostic integration with all
flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3,
and Qwen2.5-max, achieves 21.59times context compression alongside
19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct,
QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on
Ruler-128K and InfiniteBench, establishing new SOTA performance.
QwenLong-CPRS enhances large language models with multi-granularity context compression, dynamic optimization guided by natural language, and efficient bidirectional reasoning and parallel inference, achieving superior performance and context management.
context compression dynamic context optimization bidirectional reasoning layers token critic mechanisms window-parallel inference Qwen RAG sparse attention large language models SOTA performance