MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer
  Decoding
Auto-regressive inference of transformers benefit greatly from Key-Value (KV)
caching, but can lead to major memory bottlenecks as model size, batch size,
and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV)
sharing, a novel approach extending KV sharing across transformer layers to
reduce memory usage beyond what was possible with Multi-Query Attention (MQA)
and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and
inference metrics using uptrained Pythia-160M variants demonstrate that MLKV
significantly reduces memory usage with minimal performance loss, reducing KV
cache size down to a factor of 6x compared to MQA. These results highlight
MLKV's potential for efficient deployment of transformer models at scale. We
provide code at https://github.com/zaydzuhri/pythia-mlkv
666be7eef8649b8d8f562e90
Zayd Muhammad Kawakibi Zuhri
claimed_verified
2024-06-14T07:21:17.828Z
60cf8a354061635e43b28f60
https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg
Zayd Muhammad Kawakibi Zuhri
user
zaydzuhri
666be7eef8649b8d8f562e91
Muhammad Farid Adilazuarda
666be7eef8649b8d8f562e92
Ayu Purwarianti
666be7eef8649b8d8f562e93
Alham Fikri Aji
claimed_verified
2024-06-14T13:56:01.718Z
61a4dc053205e107691e0d82
https://cdn-avatars.huggingface.co/v1/production/uploads/61a4dc053205e107691e0d82/BESoEHlHYXstXudh6dOdT.jpeg
Alham Fikri Aji
user
afaji
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer
  Decoding
Auto-regressive inference of transformers benefit greatly from Key-Value (KV)
caching, but can lead to major memory bottlenecks as model size, batch size,
and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV)
sharing, a novel approach extending KV sharing across transformer layers to
reduce memory usage beyond what was possible with Multi-Query Attention (MQA)
and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and
inference metrics using uptrained Pythia-160M variants demonstrate that MLKV
significantly reduces memory usage with minimal performance loss, reducing KV
cache size down to a factor of 6x compared to MQA. These results highlight
MLKV's potential for efficient deployment of transformer models at scale. We
provide code at https://github.com/zaydzuhri/pythia-mlkv
Multi-Layer Key-Value sharing in transformers reduces memory usage without significantly affecting performance, making large-scale deployment feasible.
auto-regressive inference transformers Key-Value caching Multi-Layer Key-Value sharing Multi-Query Attention Grouped-Query Attention KV cache size NLP benchmarks