LongRoPE2: Near-Lossless LLM Context Window Scaling
LongRoPE2 is a novel approach that extends the effective context window of
pre-trained large language models (LLMs) to the target length, while preserving
the performance on the original shorter context window. This is achieved by
three contributions: (1) a hypothesis that insufficient training in higher RoPE
dimensions contributes to the persistent out-of-distribution (OOD) issues
observed in existing methods; (2) an effective RoPE rescaling algorithm that
adopts evolutionary search guided by "needle-driven" perplexity to address the
insufficient training problem; (3) a mixed context window training approach
that fine-tunes model weights to adopt rescaled RoPE for long-context sequences
while preserving the short-context performance with the original RoPE.
Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks
validate the hypothesis and demonstrate the effectiveness of LongRoPE2.
Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context
length while retaining over 98.5% of short-context performance, using only 10B
tokens -- 80x fewer than Meta's approach, which fails to reach the target
effective context length. Code will be available at
https://github.com/microsoft/LongRoPE.
67c12b6d25c74ee5b6e2ce8e
Ning Shang
admin_assigned
2025-02-28T12:28:26.117Z
632bc663eafe8eca5e9bfdbc
/avatars/787553c73e9a96adc5219e67acd29c00.svg
Ning Shang
user
J-shang
67c12b6d25c74ee5b6e2ce8f
Li Lyna Zhang
admin_assigned
2025-02-28T12:26:58.131Z
62b0009c72043b05d29492b2
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
Li Lyna Zhang
user
lynazhang
67c12b6d25c74ee5b6e2ce90
Siyuan Wang
claimed_verified
2025-02-28T12:14:20.687Z
6495b0b844bc2e9ce6cc849b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/j6aucl_tefMHwtD-bdUAw.jpeg
Siyuan Wang
user
OldKingMeister
67c12b6d25c74ee5b6e2ce91
Gaokai Zhang
admin_assigned
2025-02-28T12:28:05.040Z
65efe691ccef3501d586bb62
/avatars/c4716c532754b487359e77e43afe09bc.svg
Gaokai Zhang
user
gaokaiz2
67c12b6d25c74ee5b6e2ce92
Gilsinia Lopez
admin_assigned
2025-02-28T12:27:52.164Z
60c790f1accf7da31ed8240d
https://cdn-avatars.huggingface.co/v1/production/uploads/60c790f1accf7da31ed8240d/YDohCmgf9OUeWqZIs3Thh.jpeg
Gilsinia Lopez
user
lgg
67c12b6d25c74ee5b6e2ce93
Fan Yang
67c12b6d25c74ee5b6e2ce94
Weizhu Chen
admin_assigned
2025-02-28T12:27:17.122Z
64da876370446182be5b608d
/avatars/e412fdc71404ecdf638e416846e3ebfb.svg
Weizhu Chen
user
chenweizhu
67c12b6d25c74ee5b6e2ce95
Mao Yang
LongRoPE2: Near-Lossless LLM Context Window Scaling
LongRoPE2 is a novel approach that extends the effective context window of
pre-trained large language models (LLMs) to the target length, while preserving
the performance on the original shorter context window. This is achieved by
three contributions: (1) a hypothesis that insufficient training in higher RoPE
dimensions contributes to the persistent out-of-distribution (OOD) issues
observed in existing methods; (2) an effective RoPE rescaling algorithm that
adopts evolutionary search guided by "needle-driven" perplexity to address the
insufficient training problem; (3) a mixed context window training approach
that fine-tunes model weights to adopt rescaled RoPE for long-context sequences
while preserving the short-context performance with the original RoPE.
Extensive experiments on LLaMA3-8B and Phi3-mini-3.8B across various benchmarks
validate the hypothesis and demonstrate the effectiveness of LongRoPE2.
Remarkably, LongRoPE2 extends LLaMA3-8B to achieve a 128K effective context
length while retaining over 98.5% of short-context performance, using only 10B
tokens -- 80x fewer than Meta's approach, which fails to reach the target
effective context length. Code will be available at
https://github.com/microsoft/LongRoPE.
LongRoPE2 enhances the context window of pre-trained LLMs using rescaled RoPE and mixed context training, maintaining short-context performance with fewer tokens.
LLMs RoPE out-of-distribution evolutionary search perplexity mixed context window training effective context window