Compressing Chain-of-Thought in LLMs via Step Entropy
Large Language Models (LLMs) using Chain-of-Thought (CoT) prompting excel at
complex reasoning but generate verbose thought processes with considerable
redundancy, leading to increased inference costs and reduced efficiency. We
introduce a novel CoT compression framework based on step entropy, a metric
that quantifies the informational contribution of individual reasoning steps to
identify redundancy. Through theoretical analysis and extensive empirical
validation on mathematical reasoning benchmarks, we demonstrate that steps with
low entropy are indeed highly redundant. Our experiments reveal that an
astonishing 80\% of low-entropy intermediate steps can be pruned with minor
degradation in the final answer accuracy across DeepSeek-R1-7B, 14B and
Qwen3-8B. This finding sharply contrasts with random or high-entropy pruning,
which severely impairs reasoning performance. Building on this, we propose a
novel two-stage training strategy combining Supervised Fine-Tuning (SFT) and
Group Relative Policy Optimization (GRPO) reinforcement learning. This approach
enables LLMs to autonomously learn to generate compressed COTs during inference
by strategically incorporating [SKIP] tokens. Our method significantly enhances
LLM inference efficiency while rigorously preserving accuracy, offering
profound implications for practical LLM deployment and a deeper understanding
of reasoning structures.
689b1da6fab6fdd2e52ac686
Zeju Li
claimed_verified
2025-08-13T07:15:25.895Z
664ac4f7fe822b08e6f06814
/avatars/23193494fcc8e58faf1eee5f1223aca6.svg
Zeju Li
user
zeju-0727
689b1da6fab6fdd2e52ac687
Jianyuan Zhong
689b1da6fab6fdd2e52ac688
Ziyang Zheng
689b1da6fab6fdd2e52ac689
Xiangyu Wen
689b1da6fab6fdd2e52ac68a
Zhijian Xu
689b1da6fab6fdd2e52ac68b
Yingying Cheng
689b1da6fab6fdd2e52ac68c
Fan Zhang
689b1da6fab6fdd2e52ac68d
Qiang Xu
Compressing Chain-of-Thought in LLMs via Step Entropy
Large Language Models (LLMs) using Chain-of-Thought (CoT) prompting excel at
complex reasoning but generate verbose thought processes with considerable
redundancy, leading to increased inference costs and reduced efficiency. We
introduce a novel CoT compression framework based on step entropy, a metric
that quantifies the informational contribution of individual reasoning steps to
identify redundancy. Through theoretical analysis and extensive empirical
validation on mathematical reasoning benchmarks, we demonstrate that steps with
low entropy are indeed highly redundant. Our experiments reveal that an
astonishing 80\% of low-entropy intermediate steps can be pruned with minor
degradation in the final answer accuracy across DeepSeek-R1-7B, 14B and
Qwen3-8B. This finding sharply contrasts with random or high-entropy pruning,
which severely impairs reasoning performance. Building on this, we propose a
novel two-stage training strategy combining Supervised Fine-Tuning (SFT) and
Group Relative Policy Optimization (GRPO) reinforcement learning. This approach
enables LLMs to autonomously learn to generate compressed COTs during inference
by strategically incorporating [SKIP] tokens. Our method significantly enhances
LLM inference efficiency while rigorously preserving accuracy, offering
profound implications for practical LLM deployment and a deeper understanding
of reasoning structures.
A novel CoT compression framework using step entropy and a two-stage training strategy enhances LLM inference efficiency without significantly reducing accuracy.
Chain-of-Thought CoT prompting step entropy redundancy inference costs DeepSeek-R1-7B Qwen3-8B Supervised Fine-Tuning Group Relative Policy Optimization reinforcement learning [SKIP] tokens