Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic
  Capabilities in LLM Compression
Post-training compression reduces the computational and memory costs of large
language models (LLMs), enabling resource-efficient deployment. However,
existing compression benchmarks only focus on language modeling (e.g.,
perplexity) and natural language understanding tasks (e.g., GLUE accuracy),
ignoring the agentic capabilities - workflow, tool use/function call,
long-context understanding and real-world application. We introduce the Agent
Compression Benchmark (ACBench), the first comprehensive benchmark for
evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1)
12 tasks across 4 capabilities (e.g., WorfBench for workflow generation,
Needle-in-Haystack for long-context retrieval), (2) quantization (GPTQ, AWQ)
and pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B),
standard (Qwen2.5 7B-32B), and distilled reasoning LLMs (DeepSeek-R1-Distill).
Our experiments reveal compression tradeoffs: 4-bit quantization preserves
workflow generation and tool use (1%-3% drop) but degrades real-world
application accuracy by 10%-15%. We introduce ERank, Top-k Ranking Correlation
and Energy to systematize analysis. ACBench provides actionable insights for
optimizing LLM compression in agentic scenarios. The code can be found in
https://github.com/pprp/ACBench.
6836b4b68ec432fdc7e38251
Peijie Dong
claimed_verified
2025-06-02T07:48:54.965Z
6395f845aec00abff778ad31
https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
PeijieDong
user
pprp
6836b4b68ec432fdc7e38252
Zhenheng Tang
6836b4b68ec432fdc7e38253
Xiang Liu
claimed_verified
2025-05-28T12:12:19.286Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
6836b4b68ec432fdc7e38254
Lujun Li
6836b4b68ec432fdc7e38255
Xiaowen Chu
6836b4b68ec432fdc7e38256
Bo Li
Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic
  Capabilities in LLM Compression
Post-training compression reduces the computational and memory costs of large
language models (LLMs), enabling resource-efficient deployment. However,
existing compression benchmarks only focus on language modeling (e.g.,
perplexity) and natural language understanding tasks (e.g., GLUE accuracy),
ignoring the agentic capabilities - workflow, tool use/function call,
long-context understanding and real-world application. We introduce the Agent
Compression Benchmark (ACBench), the first comprehensive benchmark for
evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1)
12 tasks across 4 capabilities (e.g., WorfBench for workflow generation,
Needle-in-Haystack for long-context retrieval), (2) quantization (GPTQ, AWQ)
and pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B),
standard (Qwen2.5 7B-32B), and distilled reasoning LLMs (DeepSeek-R1-Distill).
Our experiments reveal compression tradeoffs: 4-bit quantization preserves
workflow generation and tool use (1%-3% drop) but degrades real-world
application accuracy by 10%-15%. We introduce ERank, Top-k Ranking Correlation
and Energy to systematize analysis. ACBench provides actionable insights for
optimizing LLM compression in agentic scenarios. The code can be found in
https://github.com/pprp/ACBench.
ACBench evaluates the impact of compression on the agentic capabilities of large language models, focusing on workflow generation, tool use, long-context understanding, and real-world application.
large language models LLMs post-training compression computational costs memory costs resource-efficient deployment language modeling perplexity natural language understanding GLUE accuracy Agent Compression Benchmark ACBench WorfBench Needle-in-Haystack quantization GPTQ AWQ pruning Wanda SparseGPT DeepSeek-R1-Distill ERank Top-k Ranking Correlation Energy agentic capabilities workflow generation tool use long-context understanding real-world application