Don't Pay Attention
The Transformer has become the de facto standard for large language models
and a wide range of downstream tasks across various domains. Despite its
numerous advantages like inherent training parallelism, the Transformer still
faces key challenges due to its inability to effectively process sequences
beyond a fixed context window and the quadratic complexity of its attention
mechanism. These challenges have renewed interest in RNN-like architectures,
which offer linear scaling with sequence length and improved handling of
long-range dependencies, albeit with limited parallelism due to their
inherently recurrent nature. In this paper, we propose Avey, a new neural
foundational architecture that breaks away from both attention and recurrence.
Avey comprises a ranker and an autoregressive neural processor, which
collaboratively identify and contextualize only the most relevant tokens for
any given token, regardless of their positions in the sequence. Specifically,
Avey decouples sequence length from context width, thus enabling effective
processing of arbitrarily long sequences. Experimental results show that Avey
compares favorably to the Transformer across a variety of standard short-range
NLP benchmarks, while notably excelling at capturing long-range dependencies.
68500ddfa81b87ad9c38629c
Mohammad Hammoud
extracted_confirmed
2025-06-16T13:37:25.460Z
68473f12c0d029ddabed529d
/avatars/2588b702884092abc7436f9df26bdfda.svg
Mohammad Hammoud
user
mhhamoud
68500ddfa81b87ad9c38629d
Devang Acharya
extracted_pending
2025-06-16T12:28:16.482Z
683c849152d744770fe82ade
/avatars/6117767025cd743c19eb1f0b3b4d3ac5.svg
Devang Acharya
user
dacharya-avey
Don't Pay Attention
The Transformer has become the de facto standard for large language models
and a wide range of downstream tasks across various domains. Despite its
numerous advantages like inherent training parallelism, the Transformer still
faces key challenges due to its inability to effectively process sequences
beyond a fixed context window and the quadratic complexity of its attention
mechanism. These challenges have renewed interest in RNN-like architectures,
which offer linear scaling with sequence length and improved handling of
long-range dependencies, albeit with limited parallelism due to their
inherently recurrent nature. In this paper, we propose Avey, a new neural
foundational architecture that breaks away from both attention and recurrence.
Avey comprises a ranker and an autoregressive neural processor, which
collaboratively identify and contextualize only the most relevant tokens for
any given token, regardless of their positions in the sequence. Specifically,
Avey decouples sequence length from context width, thus enabling effective
processing of arbitrarily long sequences. Experimental results show that Avey
compares favorably to the Transformer across a variety of standard short-range
NLP benchmarks, while notably excelling at capturing long-range dependencies.
Avey, a new neural architecture combining a ranker and an autoregressive processor, demonstrates superior performance over the Transformer in processing long-range dependencies and has competitive short-range capabilities.
Transformer RNN-like architectures attention mechanism autoregressive neural processor ranker long-range dependencies short-range NLP benchmarks context width sequence length