MoA: Mixture of Sparse Attention for Automatic Large Language Model
  Compression
Sparse attention can effectively mitigate the significant memory and
throughput demands of Large Language Models (LLMs) in long contexts. Existing
methods typically employ a uniform sparse attention mask, applying the same
sparse pattern across different attention heads and input lengths. However,
this uniform approach fails to capture the diverse attention patterns inherent
in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this
challenge, we propose the Mixture of Attention (MoA), which automatically
tailors distinct sparse attention configurations to different heads and layers.
MoA constructs and navigates a search space of various attention patterns and
their scaling rules relative to input sequence lengths. It profiles the model,
evaluates potential configurations, and pinpoints the optimal sparse attention
compression plan. MoA adapts to varying input sizes, revealing that some
attention heads expand their focus to accommodate longer sequences, while other
heads consistently concentrate on fixed-length local contexts. Experiments show
that MoA increases the effective context length by 3.9times with the same
average attention span, boosting retrieval accuracy by 1.5-7.1times over the
uniform-attention baseline across Vicuna-7B, Vicuna-13B, and Llama3-8B models.
Moreover, MoA narrows the capability gaps between sparse and dense models,
reducing the maximum relative performance drop from 9%-36% to within 5%
across two long-context understanding benchmarks. MoA achieves a
1.2-1.4times GPU memory reduction and boosts decode throughput by 5.5-6.7
times for 7B and 13B dense models on a single GPU, with minimal impact on
performance.
667e3ffc1127552103ab356e
Tianyu Fu
claimed_verified
2024-06-28T09:12:56.556Z
6445fd9ba56444c355dcbcba
https://cdn-avatars.huggingface.co/v1/production/uploads/6445fd9ba56444c355dcbcba/NCyRCD-MK-yA0_qY6I2y0.png
Tianyu Fu
user
fuvty
667e3ffc1127552103ab356f
Haofeng Huang
claimed_verified
2024-07-15T09:01:11.739Z
644950b316425b79ffee5f13
/avatars/b87d20dd659b727ec7e2d5ee4fda1852.svg
Jason Huang
user
jason-huang03
667e3ffc1127552103ab3570
Xuefei Ning
claimed_verified
2024-06-28T08:02:43.467Z
641031b1a78453b8d96b8420
https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
Xuefei Ning
user
Foxfi
667e3ffc1127552103ab3571
Genghan Zhang
admin_assigned
2024-06-28T12:32:53.942Z
65a76ff1e504d9738d636217
/avatars/26bf5e3f19057835ee95d72c24904d77.svg
Genghan Zhang
user
Genghan
667e3ffc1127552103ab3572
Boju Chen
admin_assigned
2024-06-28T12:33:02.302Z
6503e40893574a897182f27f
/avatars/2aff6e564565bd243fdb5dfb2f1ef545.svg
Boju Chen
user
Herdsman
667e3ffc1127552103ab3573
Tianqi Wu
admin_assigned
2024-06-28T12:33:08.192Z
662c3a46b7c202c0842ab92e
/avatars/d57cb6de6da34ef2b54ed434ae752d4d.svg
Tianqi Wu
user
wutianqidx
667e3ffc1127552103ab3574
Hongyi Wang
admin_assigned
2024-06-28T12:33:49.980Z
64e5414fe56d371e42da69c4
/avatars/7d4a101b0338410bc736e04d7a611aca.svg
Hongyi Wang
user
hwang595
667e3ffc1127552103ab3575
Zixiao Huang
admin_assigned
2024-06-28T12:34:01.780Z
652392b46b01183a67042fc3
/avatars/646e455e6bd39bbbf7abc8e3f3c8f225.svg
Huang Zixiao
user
Hzxx
667e3ffc1127552103ab3576
Shiyao Li
admin_assigned
2024-06-28T12:44:32.414Z
64cb1a9fb704b851478bcd5b
/avatars/b35ec1320adc058bff871f14971485ba.svg
Shiyao Li
user
LSY-noya
667e3ffc1127552103ab3577
Shengen Yan
667e3ffc1127552103ab3578
Guohao Dai
667e3ffc1127552103ab3579
Huazhong Yang
667e3ffc1127552103ab357a
Yu Wang
MoA: Mixture of Sparse Attention for Automatic Large Language Model
  Compression
Sparse attention can effectively mitigate the significant memory and
throughput demands of Large Language Models (LLMs) in long contexts. Existing
methods typically employ a uniform sparse attention mask, applying the same
sparse pattern across different attention heads and input lengths. However,
this uniform approach fails to capture the diverse attention patterns inherent
in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this
challenge, we propose the Mixture of Attention (MoA), which automatically
tailors distinct sparse attention configurations to different heads and layers.
MoA constructs and navigates a search space of various attention patterns and
their scaling rules relative to input sequence lengths. It profiles the model,
evaluates potential configurations, and pinpoints the optimal sparse attention
compression plan. MoA adapts to varying input sizes, revealing that some
attention heads expand their focus to accommodate longer sequences, while other
heads consistently concentrate on fixed-length local contexts. Experiments show
that MoA increases the effective context length by 3.9times with the same
average attention span, boosting retrieval accuracy by 1.5-7.1times over the
uniform-attention baseline across Vicuna-7B, Vicuna-13B, and Llama3-8B models.
Moreover, MoA narrows the capability gaps between sparse and dense models,
reducing the maximum relative performance drop from 9%-36% to within 5%
across two long-context understanding benchmarks. MoA achieves a
1.2-1.4times GPU memory reduction and boosts decode throughput by 5.5-6.7
times for 7B and 13B dense models on a single GPU, with minimal impact on
performance.
Mixture of Attention (MoA) adapts sparse attention configurations for different heads and layers, enhancing context length, retrieval accuracy, and throughput in Large Language Models.
sparse attention Large Language Models (LLMs) uniform sparse attention mask attention heads input lengths MoA search space attention patterns input sequence lengths effective context length retrieval accuracy GPU memory reduction decode throughput