Long-Context Attention Benchmark: From Kernel Efficiency to Distributed
  Context Parallelism
Transformer-based large language models (LLMs) have achieved remarkable
success, yet their standard attention mechanism incurs quadratic computation
and memory costs with respect to sequence length, posing a major bottleneck for
long-context training. Prior work tackles this challenge along two directions:
(1) kernel-level optimizations, which accelerate dense and sparse attention
operators; and (2) module-level strategies, often referred to as distributed
attention or context parallel training, which scale attention across multiple
devices. However, systematic evaluation still remains limited: operator-level
comparisons are often incomplete, while context parallel strategies are
typically framework-specific, with unclear performance analysis across
contexts. To address these gaps, we propose a unified benchmark that integrates
representative attention kernels and context parallel mechanisms with a modular
and extensible interface for evaluation. The benchmark evaluates methods along
two critical dimensions: (1) attention mask patterns, which strongly affect
efficiency, scalability, and usability, and (2) sequence length and distributed
scale, which determine performance under extreme long-context training. Through
comprehensive experiments on the cluster of up to 96 GPUs, our benchmark
enables reproducible comparisons, highlights method-specific trade-offs, and
provides practical guidance for designing and deploying attention mechanisms in
long-context LLM training.
68fb0671f158a71c5a2f58bd
Tao Bu
68fb0671f158a71c5a2f58be
Qiangang Wang
68fb0671f158a71c5a2f58bf
Bowen Zeng
68fb0671f158a71c5a2f58c0
Hanwen Sun
68fb0671f158a71c5a2f58c1
Yunpeng Huang
68fb0671f158a71c5a2f58c2
Chun Cao
68fb0671f158a71c5a2f58c3
Jingwei Xu
claimed_verified
2025-10-24T16:01:20.533Z
64267a357a0d3f02acd437ec
/avatars/69767c3d2b1b484403cc930f503f63e9.svg
Jingwei Xu
user
ParagonLight
Long-Context Attention Benchmark: From Kernel Efficiency to Distributed
  Context Parallelism
Transformer-based large language models (LLMs) have achieved remarkable
success, yet their standard attention mechanism incurs quadratic computation
and memory costs with respect to sequence length, posing a major bottleneck for
long-context training. Prior work tackles this challenge along two directions:
(1) kernel-level optimizations, which accelerate dense and sparse attention
operators; and (2) module-level strategies, often referred to as distributed
attention or context parallel training, which scale attention across multiple
devices. However, systematic evaluation still remains limited: operator-level
comparisons are often incomplete, while context parallel strategies are
typically framework-specific, with unclear performance analysis across
contexts. To address these gaps, we propose a unified benchmark that integrates
representative attention kernels and context parallel mechanisms with a modular
and extensible interface for evaluation. The benchmark evaluates methods along
two critical dimensions: (1) attention mask patterns, which strongly affect
efficiency, scalability, and usability, and (2) sequence length and distributed
scale, which determine performance under extreme long-context training. Through
comprehensive experiments on the cluster of up to 96 GPUs, our benchmark
enables reproducible comparisons, highlights method-specific trade-offs, and
provides practical guidance for designing and deploying attention mechanisms in
long-context LLM training.
A unified benchmark evaluates attention mechanisms in transformer-based LLMs, focusing on efficiency, scalability, and performance across different attention mask patterns and sequence lengths.
Transformer-based large language models attention mechanism quadratic computation memory costs sequence length kernel-level optimizations dense attention sparse attention distributed attention context parallel training attention mask patterns sequence length distributed scale long-context training reproducible comparisons method-specific trade-offs