Cottention: Linear Transformers With Cosine Attention
Attention mechanisms, particularly softmax attention, have been instrumental
in the success of transformer-based models such as GPT. However, the quadratic
memory complexity of softmax attention with respect to sequence length poses
significant challenges for processing longer sequences. We introduce
Cottention, a novel attention mechanism that replaces the softmax operation
with cosine similarity. By leveraging the properties of cosine similarity and
rearranging the attention equation, Cottention achieves native linear memory
complexity with respect to sequence length, making it inherently more
memory-efficient than softmax attention. We demonstrate that Cottention can be
reformulated as a recurrent neural network (RNN) with a finite hidden state,
allowing for constant memory usage during inference. We evaluate Cottention on
both the bidirectional BERT and causal GPT tasks, demonstrating comparable
performance to softmax attention while significantly reducing memory
requirements. To ensure efficient computation, we develop a custom CUDA kernel
for Cottention. Our results show that Cottention is a promising alternative to
softmax attention, enabling the processing of longer sequences without
sacrificing performance, due to its native linear memory complexity and ability
to maintain a constant memory footprint during inference.
66fac0f66cc02a6574d97326
Gabriel Mongaras
extracted_confirmed
2025-08-01T03:44:14.547Z
62a1280b88bfb47fc40fe75b
https://cdn-avatars.huggingface.co/v1/production/uploads/62a1280b88bfb47fc40fe75b/tDqTDW-XZh3n1zk3nKFxR.png
Gabriel Mongaras
user
gmongaras
66fac0f66cc02a6574d97327
Trevor Dohm
extracted_confirmed
2024-09-30T21:03:31.996Z
646a8ef31556443f24b803be
https://cdn-avatars.huggingface.co/v1/production/uploads/646a8ef31556443f24b803be/fZR4AzjlL73BVM8f60g2Q.jpeg
Trevor Dohm
user
TrevorDohm
66fac0f66cc02a6574d97328
Eric C. Larson
extracted_pending
2024-10-01T15:59:41.573Z
66fc1c6084cfed906e9aede4
/avatars/cd21024e0245ba2a02e1e3b59ada2c5c.svg
Eric Larson
user
eclarson
Cottention: Linear Transformers With Cosine Attention
Attention mechanisms, particularly softmax attention, have been instrumental
in the success of transformer-based models such as GPT. However, the quadratic
memory complexity of softmax attention with respect to sequence length poses
significant challenges for processing longer sequences. We introduce
Cottention, a novel attention mechanism that replaces the softmax operation
with cosine similarity. By leveraging the properties of cosine similarity and
rearranging the attention equation, Cottention achieves native linear memory
complexity with respect to sequence length, making it inherently more
memory-efficient than softmax attention. We demonstrate that Cottention can be
reformulated as a recurrent neural network (RNN) with a finite hidden state,
allowing for constant memory usage during inference. We evaluate Cottention on
both the bidirectional BERT and causal GPT tasks, demonstrating comparable
performance to softmax attention while significantly reducing memory
requirements. To ensure efficient computation, we develop a custom CUDA kernel
for Cottention. Our results show that Cottention is a promising alternative to
softmax attention, enabling the processing of longer sequences without
sacrificing performance, due to its native linear memory complexity and ability
to maintain a constant memory footprint during inference.
Cottention, a linear memory complexity attention mechanism based on cosine similarity, reduces memory requirements without sacrificing performance in transformer-based models.
attention mechanisms softmax attention transformer-based models Cottention cosine similarity linear memory complexity recurrent neural network finite hidden state bidirectional BERT causal GPT CUDA kernel