Mitigating Overthinking through Reasoning Shaping
Large reasoning models (LRMs) boosted by Reinforcement Learning from Verifier
Reward (RLVR) have shown great power in problem solving, yet they often cause
overthinking: excessive, meandering reasoning that inflates computational cost.
Prior designs of penalization in RLVR manage to reduce token consumption while
often harming model performance, which arises from the oversimplicity of
token-level supervision. In this paper, we argue that the granularity of
supervision plays a crucial role in balancing efficiency and accuracy, and
propose Group Relative Segment Penalization (GRSP), a step-level method to
regularize reasoning. Since preliminary analyses show that reasoning segments
are strongly correlated with token consumption and model performance, we design
a length-aware weighting mechanism across segment clusters. Extensive
experiments demonstrate that GRSP achieves superior token efficiency without
heavily compromising accuracy, especially the advantages with harder problems.
Moreover, GRSP stabilizes RL training and scales effectively across model
sizes.
68ec6e2dcd07fb414898c9a2
Feifan Song
claimed_verified
2025-10-13T16:09:22.284Z
6447ca6ca478b20f1755b294
/avatars/5049856b5ed1b74533fff902e14b4c7c.svg
Feifan Song
user
songff
68ec6e2dcd07fb414898c9a3
Shaohang Wei
claimed_verified
2025-10-13T16:09:24.252Z
67244a81aa8556c561925ab6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/w-vZ0uwYACagrNq-H1oyO.jpeg
Shaohang Wei
user
SylvainWei
68ec6e2dcd07fb414898c9a4
Bofei Gao
68ec6e2dcd07fb414898c9a5
Yejie Wang
68ec6e2dcd07fb414898c9a6
Wen Luo
68ec6e2dcd07fb414898c9a7
Wei Li
68ec6e2dcd07fb414898c9a8
Linli Yao
68ec6e2dcd07fb414898c9a9
Weimin Xiong
68ec6e2dcd07fb414898c9aa
Liang Chen
68ec6e2dcd07fb414898c9ab
Tianyu Liu
68ec6e2dcd07fb414898c9ac
Houfeng Wang
Mitigating Overthinking through Reasoning Shaping
Large reasoning models (LRMs) boosted by Reinforcement Learning from Verifier
Reward (RLVR) have shown great power in problem solving, yet they often cause
overthinking: excessive, meandering reasoning that inflates computational cost.
Prior designs of penalization in RLVR manage to reduce token consumption while
often harming model performance, which arises from the oversimplicity of
token-level supervision. In this paper, we argue that the granularity of
supervision plays a crucial role in balancing efficiency and accuracy, and
propose Group Relative Segment Penalization (GRSP), a step-level method to
regularize reasoning. Since preliminary analyses show that reasoning segments
are strongly correlated with token consumption and model performance, we design
a length-aware weighting mechanism across segment clusters. Extensive
experiments demonstrate that GRSP achieves superior token efficiency without
heavily compromising accuracy, especially the advantages with harder problems.
Moreover, GRSP stabilizes RL training and scales effectively across model
sizes.
Group Relative Segment Penalization (GRSP) improves token efficiency in large reasoning models without significantly reducing accuracy, especially for complex problems, by regularizing reasoning at the step level.
Reinforcement Learning from Verifier Reward (RLVR) overthinking token consumption model performance token-level supervision Group Relative Segment Penalization (GRSP) step-level method reasoning segments length-aware weighting mechanism segment clusters RL training model sizes