Behind RoPE: How Does Causal Mask Encode Positional Information?
While explicit positional encodings such as RoPE are a primary source of
positional information in Transformer decoders, the causal mask also provides
positional information. In this work, we prove that the causal mask can induce
position-dependent patterns in attention scores, even without parameters or
causal dependency in the input. Our theoretical analysis indicates that the
induced attention pattern tends to favor nearby query-key pairs, mirroring the
behavior of common positional encodings. Empirical analysis confirms that
trained models exhibit the same behavior, with learned parameters further
amplifying these patterns. Notably, we found that the interaction of causal
mask and RoPE distorts RoPE's relative attention score patterns into
non-relative ones. We consistently observed this effect in modern large
language models, suggesting the importance of considering the causal mask as a
source of positional information alongside explicit positional encodings.
68d60ead8ccd91bdd39ffe0d
Junu Kim
admin_assigned
2025-09-26T16:19:56.461Z
68d27e2fd44e0881a395517c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/j1Jbx1uY2-d-CKcLNqg9m.png
Junu Kim
user
JunuKim
68d60ead8ccd91bdd39ffe0e
Xiao Liu
claimed_verified
2025-09-26T12:16:33.944Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/63fb6e281b4b1bd4e7ffc5be/aiRu_bulgnxvEMrjipGoQ.jpeg
Xiao Liu
user
lx865712528
68d60ead8ccd91bdd39ffe0f
Zhenghao Lin
68d60ead8ccd91bdd39ffe10
Lei Ji
68d60ead8ccd91bdd39ffe11
Yeyun Gong
admin_assigned
2025-09-26T16:19:44.987Z
643f615aa16cd6d1f4c581de
/avatars/47753a3e82b44f81881600c52e1e8495.svg
Yeyun Gong
user
yegong
68d60ead8ccd91bdd39ffe12
Edward Choi
Behind RoPE: How Does Causal Mask Encode Positional Information?
While explicit positional encodings such as RoPE are a primary source of
positional information in Transformer decoders, the causal mask also provides
positional information. In this work, we prove that the causal mask can induce
position-dependent patterns in attention scores, even without parameters or
causal dependency in the input. Our theoretical analysis indicates that the
induced attention pattern tends to favor nearby query-key pairs, mirroring the
behavior of common positional encodings. Empirical analysis confirms that
trained models exhibit the same behavior, with learned parameters further
amplifying these patterns. Notably, we found that the interaction of causal
mask and RoPE distorts RoPE's relative attention score patterns into
non-relative ones. We consistently observed this effect in modern large
language models, suggesting the importance of considering the causal mask as a
source of positional information alongside explicit positional encodings.
The causal mask in Transformer decoders induces position-dependent attention patterns, which can interact with explicit positional encodings like RoPE, affecting their relative attention score patterns.
Transformer decoders RoPE causal mask attention scores positional encodings relative attention score patterns