Star Attention: Efficient LLM Inference over Long Sequences
Inference with Transformer-based Large Language Models (LLMs) on long
sequences is both costly and slow due to the quadratic complexity of the
self-attention mechanism. We introduce Star Attention, a two-phase block-sparse
approximation that improves computational efficiency by sharding attention
across multiple hosts while minimizing communication overhead. In the first
phase, the context is processed using blockwise-local attention across hosts,
in parallel. In the second phase, query and response tokens attend to all prior
cached tokens through sequence-global attention. Star Attention integrates
seamlessly with most Transformer-based LLMs trained with global attention,
reducing memory requirements and inference time by up to 11x while preserving
95-100% of accuracy.
67467818ee80399924ef20cc
Shantanu Acharya
claimed_verified
2024-11-27T22:20:27.963Z
626c5978f451470f86251117
/avatars/c2ecdf2d4eb8ca1c14ba427933cc38b0.svg
Shantanu Acharya
user
shantanuacharya
67467818ee80399924ef20cd
Fei Jia
67467818ee80399924ef20ce
Boris Ginsburg
Star Attention: Efficient LLM Inference over Long Sequences
Inference with Transformer-based Large Language Models (LLMs) on long
sequences is both costly and slow due to the quadratic complexity of the
self-attention mechanism. We introduce Star Attention, a two-phase block-sparse
approximation that improves computational efficiency by sharding attention
across multiple hosts while minimizing communication overhead. In the first
phase, the context is processed using blockwise-local attention across hosts,
in parallel. In the second phase, query and response tokens attend to all prior
cached tokens through sequence-global attention. Star Attention integrates
seamlessly with most Transformer-based LLMs trained with global attention,
reducing memory requirements and inference time by up to 11x while preserving
95-100% of accuracy.
Transformer-based Large Language Models (LLMs) self-attention Star Attention two-phase block-sparse approximation sharding attention communication overhead blockwise-local attention sequence sequence-global attention memory requirements inference time accuracy