Pruning the Unsurprising: Efficient Code Reasoning via First-Token
  Surprisal
Recently, Large Reasoning Models (LRMs) have demonstrated remarkable
capabilities in code reasoning by scaling up the length of Chain-of-Thought
(CoT). However, excessively long reasoning traces introduce substantial
challenges in terms of training cost, inference latency, and deployment
feasibility. While various CoT compression approaches have emerged to address
this challenge, they face inherent trade-offs: token-level methods often
disrupt syntactic and logical coherence, while step-level methods based on
perplexity fail to reliably capture the logically critical reasoning steps. In
this paper, we propose ASAP (Anchor-guided, Surprisal-based Pruning), a novel
coarse-to-fine framework for CoT compression. ASAP first performs anchor-guided
pruning to preserve the core reasoning structure, which efficiently reduces the
search space for subsequent processing. It then enables a logic-aware pruning
by selecting logically essential reasoning steps based on a novel first-token
surprisal metric. Finally, ASAP teaches models to autonomously generate and
leverage these concise CoTs at inference time, enabling efficient reasoning in
coding tasks. Experiments show that ASAP achieves state-of-the-art accuracy
across multiple code generation benchmarks while substantially reducing
training and inference costs. On the challenging LiveCodeBench v4_v5 benchmark,
our approach reduces token generation by 23.5% and inference latency by 43.5%
compared to the strongest baseline, while achieving a competitive accuracy of
36.19% in Pass@1. Our results highlight a promising direction for building
powerful and efficient LRMs.
68994f80f022d141f5d434cf
Wenhao Zeng
68994f80f022d141f5d434d0
Yaoning Wang
68994f80f022d141f5d434d1
Chao Hu
68994f80f022d141f5d434d2
Yuling Shi
claimed_verified
2025-08-11T06:45:44.939Z
645b0c3ec35da9c7afd95421
https://cdn-avatars.huggingface.co/v1/production/uploads/645b0c3ec35da9c7afd95421/vYBrCDagHsXAo6J2p-uG0.jpeg
Yuling
user
YerbaPage
68994f80f022d141f5d434d3
Chengcheng Wan
68994f80f022d141f5d434d4
Hongyu Zhang
68994f80f022d141f5d434d5
Xiaodong Gu
Pruning the Unsurprising: Efficient Code Reasoning via First-Token
  Surprisal
Recently, Large Reasoning Models (LRMs) have demonstrated remarkable
capabilities in code reasoning by scaling up the length of Chain-of-Thought
(CoT). However, excessively long reasoning traces introduce substantial
challenges in terms of training cost, inference latency, and deployment
feasibility. While various CoT compression approaches have emerged to address
this challenge, they face inherent trade-offs: token-level methods often
disrupt syntactic and logical coherence, while step-level methods based on
perplexity fail to reliably capture the logically critical reasoning steps. In
this paper, we propose ASAP (Anchor-guided, Surprisal-based Pruning), a novel
coarse-to-fine framework for CoT compression. ASAP first performs anchor-guided
pruning to preserve the core reasoning structure, which efficiently reduces the
search space for subsequent processing. It then enables a logic-aware pruning
by selecting logically essential reasoning steps based on a novel first-token
surprisal metric. Finally, ASAP teaches models to autonomously generate and
leverage these concise CoTs at inference time, enabling efficient reasoning in
coding tasks. Experiments show that ASAP achieves state-of-the-art accuracy
across multiple code generation benchmarks while substantially reducing
training and inference costs. On the challenging LiveCodeBench v4_v5 benchmark,
our approach reduces token generation by 23.5% and inference latency by 43.5%
compared to the strongest baseline, while achieving a competitive accuracy of
36.19% in Pass@1. Our results highlight a promising direction for building
powerful and efficient LRMs.
ASAP, a novel coarse-to-fine framework, compresses Chain-of-Thought in code reasoning by preserving core structure and essential steps, reducing costs and improving efficiency.
Large Reasoning Models Chain-of-Thought CoT compression anchor-guided pruning logic-aware pruning first-token surprisal metric LiveCodeBench Pass@1