Cure the headache of Transformers via Collinear Constrained Attention
As the rapid progression of practical applications based on Large Language
Models continues, the importance of extrapolating performance has grown
exponentially in the research domain. In our study, we identified an anomalous
behavior in Transformer models that had been previously overlooked, leading to
a chaos around closest tokens which carried the most important information.
We've coined this discovery the "headache of Transformers". To address this at
its core, we introduced a novel self-attention structure named Collinear
Constrained Attention (CoCA). This structure can be seamlessly integrated with
existing extrapolation, interpolation methods, and other optimization
strategies designed for traditional Transformer models. We have achieved
excellent extrapolating performance even for 16 times to 24 times of sequence
lengths during inference without any fine-tuning on our model. We have also
enhanced CoCA's computational and spatial efficiency to ensure its
practicality. We plan to open-source CoCA shortly. In the meantime, we've made
our code available in the appendix for reappearing experiments.
650903aa5ac587b3f356753f
Shiyi Zhu
admin_assigned
2023-09-20T14:26:35.255Z
65099314ed23af2c5d1e8766
/avatars/e5e61d6d937645f9ec812e23e3061fdd.svg
Shiyi Zhu
user
underskies
650903aa5ac587b3f3567540
Jing Ye
650903aa5ac587b3f3567541
Wei Jiang
650903aa5ac587b3f3567542
Qi Zhang
650903aa5ac587b3f3567543
Yifan Wu
650903aa5ac587b3f3567544
Jianguo Li
admin_assigned
2023-09-20T14:35:25.454Z
640b3ac391d9f65c58a795a4
/avatars/4c0b745b5d730cc076c798c7628f2cd2.svg
Jianguo Li
user
JungleLee
Cure the headache of Transformers via Collinear Constrained Attention
As the rapid progression of practical applications based on Large Language
Models continues, the importance of extrapolating performance has grown
exponentially in the research domain. In our study, we identified an anomalous
behavior in Transformer models that had been previously overlooked, leading to
a chaos around closest tokens which carried the most important information.
We've coined this discovery the "headache of Transformers". To address this at
its core, we introduced a novel self-attention structure named Collinear
Constrained Attention (CoCA). This structure can be seamlessly integrated with
existing extrapolation, interpolation methods, and other optimization
strategies designed for traditional Transformer models. We have achieved
excellent extrapolating performance even for 16 times to 24 times of sequence
lengths during inference without any fine-tuning on our model. We have also
enhanced CoCA's computational and spatial efficiency to ensure its
practicality. We plan to open-source CoCA shortly. In the meantime, we've made
our code available in the appendix for reappearing experiments.
A new self-attention structure called Collinear Constrained Attention (CoCA) improves the extrapolating performance of Transformer models without fine-tuning, even for significantly increased sequence lengths.
Transformer models self-attention Collinear Constrained Attention (CoCA) inference sequence lengths