Rethinking Addressing in Language Models via Contexualized Equivariant
  Positional Encoding
Transformers rely on both content-based and position-based addressing
mechanisms to make predictions, but existing positional encoding techniques
often diminish the effectiveness of position-based addressing. Many current
methods enforce rigid patterns in attention maps, limiting the ability to model
long-range dependencies and adapt to diverse tasks. Additionally, most
positional encodings are learned as general biases, lacking the specialization
required for different instances within a dataset. To address this, we propose
conTextualized equivariAnt Position
Embedding (TAPE), a novel framework that enhances
positional embeddings by incorporating sequence content across layers. TAPE
introduces dynamic, context-aware positional encodings, overcoming the
constraints of traditional fixed patterns. By enforcing permutation and
orthogonal equivariance, TAPE ensures the stability of positional encodings
during updates, improving robustness and adaptability. Our method can be easily
integrated into pre-trained transformers, offering parameter-efficient
fine-tuning with minimal overhead. Extensive experiments shows that TAPE
achieves superior performance in language modeling, arithmetic reasoning, and
long-context retrieval tasks compared to existing positional embedding
techniques.
67778d3621a9158f25a2d23b
Jiajun Zhu
claimed_verified
2025-01-03T14:02:56.780Z
65dc42ad1ed8dc6d82e3e4d8
/avatars/d2001d58b4fff24509c9e8e952042fe5.svg
Jiajun Zhu
user
lanczos
67778d3621a9158f25a2d23c
Peihao Wang
admin_assigned
2025-01-07T16:12:48.411Z
62c8f1773e28b8ee4cddca09
/avatars/9edbcc84e4ab2b335011daab8acfdbb8.svg
Peihao Wang
user
peihaowang
67778d3621a9158f25a2d23d
Ruisi Cai
admin_assigned
2025-01-07T16:14:43.111Z
63c99e2ddfac8071d01b89da
/avatars/196af8c8498aa7a930e77c0a169da678.svg
Ruisi Cai
user
CCCCRS
67778d3621a9158f25a2d23e
Jason D. Lee
extracted_pending
2025-01-03T07:09:43.441Z
64ad70f7f9f761a4c3910c63
/avatars/69b6c167299ca7a174a69ddc5a95e4a2.svg
Jason Lee
user
Jasondlee
67778d3621a9158f25a2d23f
Pan Li
extracted_pending
2025-07-28T23:05:37.001Z
68880233838fbe4aeb4f4ae6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nb6VBw_nrLNvlG8rMQ14F.png
Pan Li
user
lipan00123
67778d3621a9158f25a2d240
Zhangyang Wang
Rethinking Addressing in Language Models via Contexualized Equivariant
  Positional Encoding
Transformers rely on both content-based and position-based addressing
mechanisms to make predictions, but existing positional encoding techniques
often diminish the effectiveness of position-based addressing. Many current
methods enforce rigid patterns in attention maps, limiting the ability to model
long-range dependencies and adapt to diverse tasks. Additionally, most
positional encodings are learned as general biases, lacking the specialization
required for different instances within a dataset. To address this, we propose
conTextualized equivariAnt Position
Embedding (TAPE), a novel framework that enhances
positional embeddings by incorporating sequence content across layers. TAPE
introduces dynamic, context-aware positional encodings, overcoming the
constraints of traditional fixed patterns. By enforcing permutation and
orthogonal equivariance, TAPE ensures the stability of positional encodings
during updates, improving robustness and adaptability. Our method can be easily
integrated into pre-trained transformers, offering parameter-efficient
fine-tuning with minimal overhead. Extensive experiments shows that TAPE
achieves superior performance in language modeling, arithmetic reasoning, and
long-context retrieval tasks compared to existing positional embedding
techniques.
A novel contextualized equivariant position embedding framework, TAPE, enhances positional encodings in transformers, improving robustness and performance across language modeling, arithmetic reasoning, and long-context retrieval tasks.
Transformers content-based addressing position-based addressing positional encoding attention maps long-range dependencies dynamic positional encodings context-aware positional encodings permutation equivariance orthogonal equivariance parameter-efficient fine-tuning language modeling arithmetic reasoning long-context retrieval tasks