CLEX: Continuous Length Extrapolation for Large Language Models
Transformer-based Large Language Models (LLMs) are pioneering advances in
many natural language processing tasks, however, their exceptional capabilities
are restricted within the preset context window of Transformer. Position
Embedding (PE) scaling methods, while effective in extending the context window
to a specific length, demonstrate either notable limitations in their
extrapolation abilities or sacrificing partial performance within the context
window. Length extrapolation methods, although theoretically capable of
extending the context window beyond the training sequence length, often
underperform in practical long-context applications. To address these
challenges, we propose Continuous Length EXtrapolation (CLEX) for LLMs. We
generalise the PE scaling approaches to model the continuous dynamics by
ordinary differential equations over the length scaling factor, thereby
overcoming the constraints of current PE scaling methods designed for specific
lengths. Moreover, by extending the dynamics to desired context lengths beyond
the training sequence length, CLEX facilitates the length extrapolation with
impressive performance in practical tasks. We demonstrate that CLEX can be
seamlessly incorporated into LLMs equipped with Rotary Position Embedding, such
as LLaMA and GPT-NeoX, with negligible impact on training and inference
latency. Experimental results reveal that CLEX can effectively extend the
context window to over 4x or almost 8x training length, with no deterioration
in performance. Furthermore, when evaluated on the practical LongBench
benchmark, our model trained on a 4k length exhibits competitive performance
against state-of-the-art open-source models trained on context lengths up to
32k.
6539e1ead1ca3238d0e72c42
Guanzheng Chen
admin_assigned
2023-10-26T08:15:50.170Z
645475e2548f22be59847604
https://cdn-avatars.huggingface.co/v1/production/uploads/645475e2548f22be59847604/EhSurrZ25u31qQ2TVXQXt.jpeg
Chen
user
Guanzheng
6539e1ead1ca3238d0e72c43
Xin Li
claimed_verified
2023-10-26T07:32:27.706Z
63913b120cf6b11c487ca31d
/avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
Xin Li
user
lixin4ever
6539e1ead1ca3238d0e72c44
Zaiqiao Meng
claimed_verified
2025-08-14T13:43:50.424Z
65e58f26912c7cf158fbf37f
https://cdn-avatars.huggingface.co/v1/production/uploads/65e58f26912c7cf158fbf37f/OigSoE5-6jF9PJ18mK2Se.png
Zaiqiao Meng
user
mengzaiqiao
6539e1ead1ca3238d0e72c45
Shangsong Liang
6539e1ead1ca3238d0e72c46
Lidong Bing
admin_assigned
2023-10-26T08:16:27.010Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
CLEX: Continuous Length Extrapolation for Large Language Models
Transformer-based Large Language Models (LLMs) are pioneering advances in
many natural language processing tasks, however, their exceptional capabilities
are restricted within the preset context window of Transformer. Position
Embedding (PE) scaling methods, while effective in extending the context window
to a specific length, demonstrate either notable limitations in their
extrapolation abilities or sacrificing partial performance within the context
window. Length extrapolation methods, although theoretically capable of
extending the context window beyond the training sequence length, often
underperform in practical long-context applications. To address these
challenges, we propose Continuous Length EXtrapolation (CLEX) for LLMs. We
generalise the PE scaling approaches to model the continuous dynamics by
ordinary differential equations over the length scaling factor, thereby
overcoming the constraints of current PE scaling methods designed for specific
lengths. Moreover, by extending the dynamics to desired context lengths beyond
the training sequence length, CLEX facilitates the length extrapolation with
impressive performance in practical tasks. We demonstrate that CLEX can be
seamlessly incorporated into LLMs equipped with Rotary Position Embedding, such
as LLaMA and GPT-NeoX, with negligible impact on training and inference
latency. Experimental results reveal that CLEX can effectively extend the
context window to over 4x or almost 8x training length, with no deterioration
in performance. Furthermore, when evaluated on the practical LongBench
benchmark, our model trained on a 4k length exhibits competitive performance
against state-of-the-art open-source models trained on context lengths up to
32k.
CLEX, a method for continuous length extrapolation, extends the context window of Transformer-based LLMs beyond the training length with minimal performance loss.
Transformer-based Large Language Models Position Embedding PE scaling methods length extrapolation methods Continuous Length EXtrapolation ordinary differential equations Rotary Position Embedding LongBench benchmark