Extending LLMs' Context Window with 100 Samples
Large Language Models (LLMs) are known to have limited extrapolation ability
beyond their pre-trained context window, constraining their application in
downstream tasks with lengthy inputs. Recent studies have sought to extend
LLMs' context window by modifying rotary position embedding (RoPE), a popular
position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and
GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are
resource-intensive and lack comparative experiments to assess their
applicability. In this work, we identify the inherent need for LLMs' attention
entropy (i.e. the information entropy of attention scores) to maintain
stability and introduce a novel extension to RoPE which combines adjusting
RoPE's base frequency and scaling the attention logits to help LLMs efficiently
adapt to a larger context window. We validate the superiority of our method in
both fine-tuning performance and robustness across different context window
sizes on various context-demanding tasks. Notably, our method extends the
context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6
training steps, showcasing extraordinary efficiency. Finally, we also explore
how data compositions and training curricula affect context window extension
for specific downstream tasks, suggesting fine-tuning LLMs with lengthy
conversations as a good starting point. We release our code and SFT data at
https://github.com/GAIR-NLP/Entropy-ABF.
65a77c2921943858bde10da1
Yikai Zhang
admin_assigned
2024-01-17T09:44:57.954Z
63e8b792ca4fc7d30de6975b
/avatars/57237f54d61d479df15209497a3f531e.svg
Yikai Zhang
user
Arist12
65a77c2921943858bde10da2
Junlong Li
admin_assigned
2024-01-17T09:46:21.036Z
621e40ac944c7e36aaec2369
https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
Junlong Li
user
lockon
65a77c2921943858bde10da3
Pengfei Liu
admin_assigned
2024-01-17T09:46:04.774Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
Extending LLMs' Context Window with 100 Samples
Large Language Models (LLMs) are known to have limited extrapolation ability
beyond their pre-trained context window, constraining their application in
downstream tasks with lengthy inputs. Recent studies have sought to extend
LLMs' context window by modifying rotary position embedding (RoPE), a popular
position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and
GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are
resource-intensive and lack comparative experiments to assess their
applicability. In this work, we identify the inherent need for LLMs' attention
entropy (i.e. the information entropy of attention scores) to maintain
stability and introduce a novel extension to RoPE which combines adjusting
RoPE's base frequency and scaling the attention logits to help LLMs efficiently
adapt to a larger context window. We validate the superiority of our method in
both fine-tuning performance and robustness across different context window
sizes on various context-demanding tasks. Notably, our method extends the
context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6
training steps, showcasing extraordinary efficiency. Finally, we also explore
how data compositions and training curricula affect context window extension
for specific downstream tasks, suggesting fine-tuning LLMs with lengthy
conversations as a good starting point. We release our code and SFT data at
https://github.com/GAIR-NLP/Entropy-ABF.
A novel extension to rotary position embedding, adjusted by modifying the base frequency and scaling attention logits, enhances LLMs' context window efficiently and robustly across various tasks.
rotary position embedding attention entropy attention scores Position Interpolation (PI) YaRN attention logits LLaMA PaLM GPT-NeoX fine-tuning performance data compositions training curricula LLaMA-2-7B-Chat