Sparse Query Attention (SQA): A Computationally Efficient Attention
  Mechanism with Query Heads Reduction
The Transformer architecture, underpinned by the Multi-Head Attention (MHA)
mechanism, has become the de facto standard for state-of-the-art models in
artificial intelligence. However, the quadratic computational complexity of MHA
with respect to sequence length presents a significant barrier to scaling,
particularly for applications involving long contexts. Prevailing solutions,
such as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), have
effectively addressed the memory bandwidth bottleneck that dominates
autoregressive inference latency by sharing Key and Value projections. While
highly successful, these methods do not reduce the fundamental number of
floating-point operations (FLOPs) required for the attention score computation,
which remains a critical bottleneck for training and full-sequence processing.
This paper introduces Sparse Query Attention (SQA), a novel attention
architecture that pursues an alternative and complementary optimization path.
Instead of reducing Key/Value heads, SQA reduces the number of Query heads.
This architectural modification directly decreases the computational complexity
of the attention mechanism by a factor proportional to the reduction in query
heads, thereby lowering the overall FLOPs. This work presents the theoretical
foundation of SQA, its mathematical formulation, and a family of architectural
variants. Empirical benchmarks on long sequences (32k-200k tokens) demonstrate
that SQA can achieve significant throughput improvements of up to 3x in
computation-bound scenarios such as model pre-training, fine-tuning, and
encoder-based tasks, with only a minimal impact on model quality in preliminary
smallscale experiments. SQA was discovered serendipitously during the
development of the upcoming Reactive Transformer architecture, suggesting its
potential as a powerful tool for building more efficient and scalable models
68dfacffdf49fb0df1e03ec3
Adam Filipek
claimed_verified
2025-10-03T12:28:11.037Z
675197c3ae96d7ba4b4a6c66
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/I2GHgrv70cfT8C5EbK6Q5.png
Adam Filipek
user
AdamF92
Sparse Query Attention (SQA): A Computationally Efficient Attention
  Mechanism with Query Heads Reduction
The Transformer architecture, underpinned by the Multi-Head Attention (MHA)
mechanism, has become the de facto standard for state-of-the-art models in
artificial intelligence. However, the quadratic computational complexity of MHA
with respect to sequence length presents a significant barrier to scaling,
particularly for applications involving long contexts. Prevailing solutions,
such as Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), have
effectively addressed the memory bandwidth bottleneck that dominates
autoregressive inference latency by sharing Key and Value projections. While
highly successful, these methods do not reduce the fundamental number of
floating-point operations (FLOPs) required for the attention score computation,
which remains a critical bottleneck for training and full-sequence processing.
This paper introduces Sparse Query Attention (SQA), a novel attention
architecture that pursues an alternative and complementary optimization path.
Instead of reducing Key/Value heads, SQA reduces the number of Query heads.
This architectural modification directly decreases the computational complexity
of the attention mechanism by a factor proportional to the reduction in query
heads, thereby lowering the overall FLOPs. This work presents the theoretical
foundation of SQA, its mathematical formulation, and a family of architectural
variants. Empirical benchmarks on long sequences (32k-200k tokens) demonstrate
that SQA can achieve significant throughput improvements of up to 3x in
computation-bound scenarios such as model pre-training, fine-tuning, and
encoder-based tasks, with only a minimal impact on model quality in preliminary
smallscale experiments. SQA was discovered serendipitously during the
development of the upcoming Reactive Transformer architecture, suggesting its
potential as a powerful tool for building more efficient and scalable models
Sparse Query Attention (SQA) reduces computational complexity in Transformer models by decreasing the number of Query heads, leading to significant throughput improvements with minimal impact on model quality.
Transformer architecture Multi-Head Attention (MHA) Multi-Query Attention (MQA) Grouped-Query Attention (GQA) Sparse Query Attention (SQA) floating-point operations (FLOPs) attention score computation Reactive Transformer architecture