Train Long, Think Short: Curriculum Learning for Efficient Reasoning
Recent work on enhancing the reasoning abilities of large language models
(LLMs) has introduced explicit length control as a means of constraining
computational cost while preserving accuracy. However, existing approaches rely
on fixed-length training budgets, which do not take advantage of the natural
progression from exploration to compression during learning. In this work, we
propose a curriculum learning strategy for length-controlled reasoning using
Group Relative Policy Optimization (GRPO). Our method starts with generous
token budgets and gradually tightens them over training, encouraging models to
first discover effective solution strategies and then distill them into more
concise reasoning traces. We augment GRPO with a reward function that balances
three signals: task correctness (via verifier feedback), length efficiency, and
formatting adherence (via structural tags). Experiments on GSM8K, MATH500,
SVAMP, College Math, and GSM+ demonstrate that curriculum-based training
consistently outperforms fixed-budget baselines at the same final budget,
achieving higher accuracy and significantly improved token efficiency. We
further ablate the impact of reward weighting and decay schedule design,
showing that progressive constraint serves as a powerful inductive bias for
training efficient reasoning models. Our code and checkpoints are released at:
https://github.com/hammoudhasan/curriculum_grpo.
689c3f0cfab6fdd2e52ac93a
Hasan Abed Al Kader Hammoud
claimed_verified
2025-08-14T13:38:43.096Z
642b51385bf2355d02a23d15
/avatars/87985347643b2647555f2453fa4d94fb.svg
Hasan Abed Al Kader Hammoud
user
hammh0a
689c3f0cfab6fdd2e52ac93b
Kumail Alhamoud
689c3f0cfab6fdd2e52ac93c
Abed Hammoud
689c3f0cfab6fdd2e52ac93d
Elie Bou-Zeid
689c3f0cfab6fdd2e52ac93e
Marzyeh Ghassemi
689c3f0cfab6fdd2e52ac93f
Bernard Ghanem
Train Long, Think Short: Curriculum Learning for Efficient Reasoning
Recent work on enhancing the reasoning abilities of large language models
(LLMs) has introduced explicit length control as a means of constraining
computational cost while preserving accuracy. However, existing approaches rely
on fixed-length training budgets, which do not take advantage of the natural
progression from exploration to compression during learning. In this work, we
propose a curriculum learning strategy for length-controlled reasoning using
Group Relative Policy Optimization (GRPO). Our method starts with generous
token budgets and gradually tightens them over training, encouraging models to
first discover effective solution strategies and then distill them into more
concise reasoning traces. We augment GRPO with a reward function that balances
three signals: task correctness (via verifier feedback), length efficiency, and
formatting adherence (via structural tags). Experiments on GSM8K, MATH500,
SVAMP, College Math, and GSM+ demonstrate that curriculum-based training
consistently outperforms fixed-budget baselines at the same final budget,
achieving higher accuracy and significantly improved token efficiency. We
further ablate the impact of reward weighting and decay schedule design,
showing that progressive constraint serves as a powerful inductive bias for
training efficient reasoning models. Our code and checkpoints are released at:
https://github.com/hammoudhasan/curriculum_grpo.
A curriculum learning strategy using Group Relative Policy Optimization (GRPO) enhances the reasoning abilities of large language models by progressively tightening token budgets, improving accuracy and token efficiency.
Group Relative Policy Optimization (GRPO) curriculum learning token budgets verifier feedback structural tags GSM8K MATH500 SVAMP College Math GSM+