SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation
Key-Value (KV) cache has become a bottleneck of LLMs for long-context
generation. Despite the numerous efforts in this area, the optimization for the
decoding phase is generally ignored. However, we believe such optimization is
crucial, especially for long-output generation tasks based on the following two
observations: (i) Excessive compression during the prefill phase, which
requires specific full context impairs the comprehension of the reasoning task;
(ii) Deviation of heavy hitters occurs in the reasoning tasks with long
outputs. Therefore, SCOPE, a simple yet efficient framework that separately
performs KV cache optimization during the prefill and decoding phases, is
introduced. Specifically, the KV cache during the prefill phase is preserved to
maintain the essential information, while a novel strategy based on sliding is
proposed to select essential heavy hitters for the decoding phase. Memory usage
and memory transfer are further optimized using adaptive and discontinuous
strategies. Extensive experiments on LongGenBench show the effectiveness and
generalization of SCOPE and its compatibility as a plug-in to other
prefill-only KV compression methods.
6768cd61aa9027defefa2ad4
Jialong Wu
claimed_verified
2024-12-23T11:10:00.992Z
644a4fbc2166258fccc664bc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8k3b44MbhQiWuo6i8BnYl.jpeg
Jialong Wu
user
callanwu
6768cd61aa9027defefa2ad5
Zhenglin Wang
claimed_verified
2024-12-30T19:36:19.930Z
6643261b8876db14227eeb19
/avatars/67428c9e37a2273697c0547e1783ec6b.svg
Zhenglin Wang
user
wzl0228
6768cd61aa9027defefa2ad6
Linhai Zhang
admin_assigned
2024-12-23T11:30:58.318Z
66596d64ce1b2838888f4401
/avatars/d8d0d116a3198571c7e86f09871c2d76.svg
Linhai Zhang
user
lzhang472
6768cd61aa9027defefa2ad7
Yilong Lai
6768cd61aa9027defefa2ad8
Yulan He
6768cd61aa9027defefa2ad9
Deyu Zhou
admin_assigned
2024-12-23T11:31:24.902Z
64e821f2bddc5b1072b15c2e
/avatars/618b5a48f2fa62daff4e1922a9aa9e8b.svg
zhoudeyu
user
zhoudeyu
SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation
Key-Value (KV) cache has become a bottleneck of LLMs for long-context
generation. Despite the numerous efforts in this area, the optimization for the
decoding phase is generally ignored. However, we believe such optimization is
crucial, especially for long-output generation tasks based on the following two
observations: (i) Excessive compression during the prefill phase, which
requires specific full context impairs the comprehension of the reasoning task;
(ii) Deviation of heavy hitters occurs in the reasoning tasks with long
outputs. Therefore, SCOPE, a simple yet efficient framework that separately
performs KV cache optimization during the prefill and decoding phases, is
introduced. Specifically, the KV cache during the prefill phase is preserved to
maintain the essential information, while a novel strategy based on sliding is
proposed to select essential heavy hitters for the decoding phase. Memory usage
and memory transfer are further optimized using adaptive and discontinuous
strategies. Extensive experiments on LongGenBench show the effectiveness and
generalization of SCOPE and its compatibility as a plug-in to other
prefill-only KV compression methods.
SCOPE is a framework that optimizes KV cache separately during prefill and decoding phases to improve long-context and long-output generation in LLMs.
KV cache LLMs long-context generation optimization decoding phase prefill phase full context heavy hitters LongGenBench adaptive strategies discontinuous strategies