The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM
  Compression Preserve?
Motivated by reducing the computational and storage costs of LLMs, model
compression and KV cache compression have attracted much attention from
researchers. However, current methods predominantly emphasize maintaining the
performance of compressed LLMs, as measured by perplexity or simple accuracy on
tasks of common sense knowledge QA and basic arithmetic reasoning. In this
blog, we present a brief review of recent advancements in LLMs related to
retrieval-augmented generation, multi-step reasoning, external tools, and
computational expressivity, all of which substantially enhance LLM performance.
Then, we propose a lottery LLM hypothesis suggesting that for a given LLM and
task, there exists a smaller lottery LLM capable of producing the same
performance as the original LLM with the assistance of multi-step reasoning and
external tools. Based on the review of current progress in LLMs, we discuss and
summarize the essential capabilities that the lottery LLM and KV cache
compression must possess, which are currently overlooked in existing methods.
67beaec94a1d9d7e368a7840
Zhenheng Tang
admin_assigned
2025-02-26T09:15:34.971Z
66a4a319a1711696948b045c
/avatars/1d92d57a949332cb8227697b9a0c2f39.svg
Zhenheng Tang
user
coolzhtang
67beaec94a1d9d7e368a7841
Xiang Liu
claimed_verified
2025-02-27T09:15:20.100Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
67beaec94a1d9d7e368a7842
Qian Wang
67beaec94a1d9d7e368a7843
Peijie Dong
claimed_verified
2025-06-02T07:52:55.355Z
6395f845aec00abff778ad31
https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
PeijieDong
user
pprp
67beaec94a1d9d7e368a7844
Bingsheng He
67beaec94a1d9d7e368a7845
Xiaowen Chu
admin_assigned
2025-02-26T09:15:41.884Z
6676935fcd0b89a0115174b0
/avatars/4caca1b672d29e787814f9a30bf20bcc.svg
Xiaowen Chu
user
wenxinsiju
67beaec94a1d9d7e368a7846
Bo Li
The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM
  Compression Preserve?
Motivated by reducing the computational and storage costs of LLMs, model
compression and KV cache compression have attracted much attention from
researchers. However, current methods predominantly emphasize maintaining the
performance of compressed LLMs, as measured by perplexity or simple accuracy on
tasks of common sense knowledge QA and basic arithmetic reasoning. In this
blog, we present a brief review of recent advancements in LLMs related to
retrieval-augmented generation, multi-step reasoning, external tools, and
computational expressivity, all of which substantially enhance LLM performance.
Then, we propose a lottery LLM hypothesis suggesting that for a given LLM and
task, there exists a smaller lottery LLM capable of producing the same
performance as the original LLM with the assistance of multi-step reasoning and
external tools. Based on the review of current progress in LLMs, we discuss and
summarize the essential capabilities that the lottery LLM and KV cache
compression must possess, which are currently overlooked in existing methods.
A proposal for a lottery LLM hypothesis to reduce the size of LLMs with similar performance using multi-step reasoning and external tools, while identifying essential capabilities that current model and cache compression methods overlook.
LLM compression KV cache compression retrieval-augmented generation multi-step reasoning external tools computational expressivity lottery LLM hypothesis