When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context
  Training
Extending context window sizes allows large language models (LLMs) to process
longer sequences and handle more complex tasks. Rotary Positional Embedding
(RoPE) has become the de facto standard due to its relative positional encoding
properties that benefit long-context training. However, we observe that using
RoPE with BFloat16 format results in numerical issues, causing it to deviate
from its intended relative positional encoding, especially in long-context
scenarios. This issue arises from BFloat16's limited precision and accumulates
as context length increases, with the first token contributing significantly to
this problem. To address this, we develop AnchorAttention, a plug-and-play
attention method that alleviates numerical issues caused by BFloat16, improves
long-context capabilities, and speeds up training. AnchorAttention reduces
unnecessary attention computations, maintains semantic coherence, and boosts
computational efficiency by treating the first token as a shared anchor with a
consistent position ID, making it visible to all documents within the training
context. Experiments on three types of LLMs demonstrate that AnchorAttention
significantly improves long-context performance and reduces training time by
over 50\% compared to standard full attention mechanisms, while preserving the
original LLM's capabilities on general tasks. Our code is available at
https://github.com/haonan3/AnchorContext.
673ea9f174d6cdda59f38d4d
Haonan Wang
claimed_verified
2024-11-21T08:09:35.543Z
6496b06a4a9a7e1fe4253ae2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/144NlRW_ETmmOgSYUs_SM.png
Haonan Wang
user
haonan3
673ea9f174d6cdda59f38d4e
Qian Liu
claimed_verified
2024-11-21T21:33:06.403Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
673ea9f174d6cdda59f38d4f
Chao Du
673ea9f174d6cdda59f38d50
Tongyao Zhu
claimed_verified
2025-03-25T08:22:26.559Z
647db42711084fb58318b053
/avatars/901a4db8e99f6e0c7e7e596c251b315c.svg
Tongyao
user
tyzhu
673ea9f174d6cdda59f38d51
Cunxiao Du
673ea9f174d6cdda59f38d52
Kenji Kawaguchi
673ea9f174d6cdda59f38d53
Tianyu Pang
claimed_verified
2025-04-20T15:04:58.424Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context
  Training
Extending context window sizes allows large language models (LLMs) to process
longer sequences and handle more complex tasks. Rotary Positional Embedding
(RoPE) has become the de facto standard due to its relative positional encoding
properties that benefit long-context training. However, we observe that using
RoPE with BFloat16 format results in numerical issues, causing it to deviate
from its intended relative positional encoding, especially in long-context
scenarios. This issue arises from BFloat16's limited precision and accumulates
as context length increases, with the first token contributing significantly to
this problem. To address this, we develop AnchorAttention, a plug-and-play
attention method that alleviates numerical issues caused by BFloat16, improves
long-context capabilities, and speeds up training. AnchorAttention reduces
unnecessary attention computations, maintains semantic coherence, and boosts
computational efficiency by treating the first token as a shared anchor with a
consistent position ID, making it visible to all documents within the training
context. Experiments on three types of LLMs demonstrate that AnchorAttention
significantly improves long-context performance and reduces training time by
over 50\% compared to standard full attention mechanisms, while preserving the
original LLM's capabilities on general tasks. Our code is available at
https://github.com/haonan3/AnchorContext.
AnchorAttention improves the long-context performance and training efficiency of large language models by addressing numerical issues with BFloat16 and Rotary Positional Embedding.
large language models context window sizes Rotary Positional Embedding BFloat16 numerical issues relative positional encoding long-context training AnchorAttention attention method semantic coherence computational efficiency shared anchor position ID full attention mechanisms