Efficient Inference of Vision Instruction-Following Models with Elastic
  Cache
In the field of instruction-following large vision-language models (LVLMs),
the efficient deployment of these models faces challenges, notably due to the
high memory demands of their key-value (KV) caches. Conventional cache
management strategies for LLMs focus on cache eviction, which often fails to
address the specific needs of multimodal instruction-following models.
Recognizing this gap, in this paper, we introduce Elastic Cache, a novel
approach that benefits from applying distinct acceleration methods for
instruction encoding and output generation stages. We investigate the metrics
of importance in different stages and propose an importance-driven cache
merging strategy to prune redundancy caches. Instead of discarding less
important caches, our strategy identifies important key/value vectors as anchor
points. Surrounding less important caches are then merged with these anchors,
enhancing the preservation of contextual information in the KV caches while
yielding an arbitrary acceleration ratio. For instruction encoding, we utilize
the frequency to evaluate the importance of caches. Regarding output
generation, we prioritize tokens based on their distance with an offset, by
which both the initial and most recent tokens are retained. Results on a range
of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also
notably outperforms existing pruning methods in language generation across
various tasks. Code is available at https://github.com/liuzuyan/ElasticCache
66a3aa35ee3de8c56edca432
Zuyan Liu
admin_assigned
2024-07-26T14:03:37.394Z
64f001bfabd9fb1914398bd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
liuzuyan
user
Zuyan
66a3aa35ee3de8c56edca433
Benlin Liu
claimed_verified
2024-08-02T12:23:18.285Z
6326723c21c98f83d6ad128e
/avatars/6c208ff815c037d0a6a2ada3198e7fd4.svg
Benlin Liu
user
Tim666
66a3aa35ee3de8c56edca434
Jiahui Wang
66a3aa35ee3de8c56edca435
Yuhao Dong
admin_assigned
2024-07-26T14:06:03.783Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
66a3aa35ee3de8c56edca436
Guangyi Chen
admin_assigned
2024-07-26T14:06:11.477Z
64975e18a08b5c7ea0163be7
/avatars/88963b6ae00ec3f10f4a40322f75944a.svg
Guangyi Chen
user
CHENGY12
66a3aa35ee3de8c56edca437
Yongming Rao
admin_assigned
2024-07-26T14:06:19.070Z
63e4865354f51ea342d45d78
/avatars/2e7eccc878751331ca8b282f53e38899.svg
Yongming Rao
user
raoyongming
66a3aa35ee3de8c56edca438
Ranjay Krishna
admin_assigned
2024-07-26T14:06:25.297Z
66429868ab89e3a3a85668b0
/avatars/170e0daa454838deee2bf946f7118651.svg
Ranjay Krishna
user
ranjaykrishna
66a3aa35ee3de8c56edca439
Jiwen Lu
Efficient Inference of Vision Instruction-Following Models with Elastic
  Cache
In the field of instruction-following large vision-language models (LVLMs),
the efficient deployment of these models faces challenges, notably due to the
high memory demands of their key-value (KV) caches. Conventional cache
management strategies for LLMs focus on cache eviction, which often fails to
address the specific needs of multimodal instruction-following models.
Recognizing this gap, in this paper, we introduce Elastic Cache, a novel
approach that benefits from applying distinct acceleration methods for
instruction encoding and output generation stages. We investigate the metrics
of importance in different stages and propose an importance-driven cache
merging strategy to prune redundancy caches. Instead of discarding less
important caches, our strategy identifies important key/value vectors as anchor
points. Surrounding less important caches are then merged with these anchors,
enhancing the preservation of contextual information in the KV caches while
yielding an arbitrary acceleration ratio. For instruction encoding, we utilize
the frequency to evaluate the importance of caches. Regarding output
generation, we prioritize tokens based on their distance with an offset, by
which both the initial and most recent tokens are retained. Results on a range
of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also
notably outperforms existing pruning methods in language generation across
various tasks. Code is available at https://github.com/liuzuyan/ElasticCache
Elastic Cache is introduced to enhance the efficiency of instruction-following large vision-language models by applying importance-driven cache merging strategies in specific stages of the model.
key-value caches cache eviction Elastic Cache acceleration methods importance-driven cache merging multimodal instruction-following models LVLMs instruction encoding output generation frequency evaluation token distance language generation