Fourier Position Embedding: Enhancing Attention's Periodic Extension for
  Length Generalization
Extending the context length of Language Models (LMs) by improving Rotary
Position Embedding (RoPE) has become a trend. While existing works mainly
address RoPE's limitations within attention mechanism, this paper provides an
analysis across nearly all parts of LMs, uncovering their adverse effects on
length generalization for RoPE-based attention. Using Discrete Signal
Processing theory, we show that RoPE enables periodic attention by implicitly
achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is
undermined by the spectral damage caused by: 1) linear layers and activation
functions outside of attention; 2) insufficiently trained frequency components
brought by time-domain truncation. Building on our observations, we propose
Fourier Position Embedding (FoPE), which enhances attention's frequency-domain
properties to improve both its periodic extension and length generalization.
FoPE constructs Fourier Series and zero-outs the destructive frequency
components, increasing model robustness against the spectrum damage.
Experiments across various model scales show that, within varying context
windows, FoPE can maintain a more stable perplexity and a more consistent
accuracy in a needle-in-haystack task compared to RoPE and ALiBi. Several
analyses and ablations bring further support to our method and theoretical
modeling.
676a6844bee647b8c004f469
Ermo Hua
claimed_verified
2024-12-30T19:34:47.877Z
6445fa2ffc22e309d78bef3e
https://cdn-avatars.huggingface.co/v1/production/uploads/6445fa2ffc22e309d78bef3e/FQaINLd0PjgY9EnK_APRk.jpeg
Messi Hua
user
Messi-Hua
676a6844bee647b8c004f46a
Che Jiang
676a6844bee647b8c004f46b
Xingtai Lv
claimed_verified
2025-03-17T08:45:51.765Z
663f07d029be04778ba97871
/avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
Xingtai Lv
user
XingtaiHF
676a6844bee647b8c004f46c
Kaiyan Zhang
claimed_verified
2024-12-30T19:34:45.610Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
676a6844bee647b8c004f46d
Ning Ding
claimed_verified
2025-01-31T09:50:47.792Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
676a6844bee647b8c004f46e
Youbang Sun
676a6844bee647b8c004f46f
Biqing Qi
676a6844bee647b8c004f470
Yuchen Fan
676a6844bee647b8c004f471
Xue Kai Zhu
676a6844bee647b8c004f472
Bowen Zhou
Fourier Position Embedding: Enhancing Attention's Periodic Extension for
  Length Generalization
Extending the context length of Language Models (LMs) by improving Rotary
Position Embedding (RoPE) has become a trend. While existing works mainly
address RoPE's limitations within attention mechanism, this paper provides an
analysis across nearly all parts of LMs, uncovering their adverse effects on
length generalization for RoPE-based attention. Using Discrete Signal
Processing theory, we show that RoPE enables periodic attention by implicitly
achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is
undermined by the spectral damage caused by: 1) linear layers and activation
functions outside of attention; 2) insufficiently trained frequency components
brought by time-domain truncation. Building on our observations, we propose
Fourier Position Embedding (FoPE), which enhances attention's frequency-domain
properties to improve both its periodic extension and length generalization.
FoPE constructs Fourier Series and zero-outs the destructive frequency
components, increasing model robustness against the spectrum damage.
Experiments across various model scales show that, within varying context
windows, FoPE can maintain a more stable perplexity and a more consistent
accuracy in a needle-in-haystack task compared to RoPE and ALiBi. Several
analyses and ablations bring further support to our method and theoretical
modeling.
Rotary Position Embedding (RoPE) attention mechanism length generalization Discrete Signal Processing theory periodic attention Non-Uniform Discrete Fourier Transform spectral damage linear layers activation functions frequency components time-domain truncation Fourier Position Embedding (FoPE) frequency-domain properties periodic extension Fourier Series destructive frequency components model robustness perplexity needle-in-haystack task ALiBi