Extending Context Window of Large Language Models via Positional
  Interpolation
We present Position Interpolation (PI) that extends the context window sizes
of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal
fine-tuning (within 1000 steps), while demonstrating strong empirical results
on various tasks that require long context, including passkey retrieval,
language modeling, and long document summarization from LLaMA 7B to 65B.
Meanwhile, the extended model by Position Interpolation preserve quality
relatively well on tasks within its original context window. To achieve this
goal, Position Interpolation linearly down-scales the input position indices to
match the original context window size, rather than extrapolating beyond the
trained context length which may lead to catastrophically high attention scores
that completely ruin the self-attention mechanism. Our theoretical study shows
that the upper bound of interpolation is at least sim 600 times smaller
than that of extrapolation, further demonstrating its stability. Models
extended via Position Interpolation retain its original architecture and can
reuse most pre-existing optimization and infrastructure.
649b963fb45a4d4bd3ae2deb
Shouyuan Chen
649b963fb45a4d4bd3ae2dec
Sherman Wong
649b963fb45a4d4bd3ae2ded
Liangjian Chen
649b963fb45a4d4bd3ae2dee
Yuandong Tian
claimed_verified
2023-06-28T20:16:00.161Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
Extending Context Window of Large Language Models via Positional
  Interpolation
We present Position Interpolation (PI) that extends the context window sizes
of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal
fine-tuning (within 1000 steps), while demonstrating strong empirical results
on various tasks that require long context, including passkey retrieval,
language modeling, and long document summarization from LLaMA 7B to 65B.
Meanwhile, the extended model by Position Interpolation preserve quality
relatively well on tasks within its original context window. To achieve this
goal, Position Interpolation linearly down-scales the input position indices to
match the original context window size, rather than extrapolating beyond the
trained context length which may lead to catastrophically high attention scores
that completely ruin the self-attention mechanism. Our theoretical study shows
that the upper bound of interpolation is at least sim 600 times smaller
than that of extrapolation, further demonstrating its stability. Models
extended via Position Interpolation retain its original architecture and can
reuse most pre-existing optimization and infrastructure.
Position Interpolation extends the context window of RoPE-based LLMs up to 32768 tokens with minimal fine-tuning, maintaining task performance and stability compared to extrapolation methods.
Position Interpolation RoPE-based pretrained LLMs context window fine-tuning passkey retrieval language modeling long document summarization LLaMA 7B LLaMA 65B input position indices self-attention mechanism extrapolation