Breaking the Attention Bottleneck
Attention-based transformers have become the standard architecture in many
deep learning fields, primarily due to their ability to model long-range
dependencies and handle variable-length input sequences. However, the attention
mechanism with its quadratic complexity is a significant bottleneck in the
transformer architecture. This algorithm is only uni-directional in the decoder
and converges to a static pattern in over-parametrized decoder-only models. I
address this issue by developing a generative function as attention or
activation replacement. It still has the auto-regressive character by comparing
each token with the previous one. In my test setting with nanoGPT this yields a
smaller loss while having a smaller model. The loss further drops by
incorporating an average context vector. This concept of attention replacement
is distributed under the GNU AGPL v3 license at
https://gitlab.com/Bachstelze/causal_generation.
6671219645b689bdb3f8ef60
Kalle Hilsenbek
claimed_verified
2024-06-18T07:17:34.044Z
62d1ddfac58f969c1528f1b5
/avatars/75c372a831cde3c7c6dce3bc875488a7.svg
Kalle Hilsenbek
user
Bachstelze
Breaking the Attention Bottleneck
Attention-based transformers have become the standard architecture in many
deep learning fields, primarily due to their ability to model long-range
dependencies and handle variable-length input sequences. However, the attention
mechanism with its quadratic complexity is a significant bottleneck in the
transformer architecture. This algorithm is only uni-directional in the decoder
and converges to a static pattern in over-parametrized decoder-only models. I
address this issue by developing a generative function as attention or
activation replacement. It still has the auto-regressive character by comparing
each token with the previous one. In my test setting with nanoGPT this yields a
smaller loss while having a smaller model. The loss further drops by
incorporating an average context vector. This concept of attention replacement
is distributed under the GNU AGPL v3 license at
https://gitlab.com/Bachstelze/causal_generation.
A new generative function replaces the attention mechanism in transformers, reducing complexity and loss while maintaining auto-regressive properties.
attention-based transformers long-range dependencies variable-length input sequences quadratic complexity uni-directional over-parametrized generative function auto-regressive nanoGPT average context vector causal_generation