CurES: From Gradient Analysis to Efficient Curriculum Learning for
  Reasoning LLMs
Curriculum learning plays a crucial role in enhancing the training efficiency
of large language models (LLMs) on reasoning tasks. However, existing methods
often fail to adequately account for variations in prompt difficulty or rely on
simplistic filtering mechanisms to select prompt datasets within a narrow
criterion range, resulting in significant computational waste. In this work, we
approach the problem from the perspective of reinforcement learning gradient
optimization, offering a systematic and theoretical investigation into how to
improve the training efficiency of LLMs. We identify two key factors
influencing training efficiency: the selection of training prompts and the
allocation of rollout quantities across different prompts. Our theoretical
analysis reveals that the sampling distribution of prompts dictates the
convergence rate of gradient descent, while the allocation of the rollout
quantity influences the consistency and stability of overall gradient updates.
Based on these insights, we propose CurES, an efficient training method that
accelerates convergence and employs Bayesian posterior estimation to minimize
computational overhead. Experiments demonstrate that our CurES outperforms
Group Relative Policy Optimization (GRPO) by +3.30 points and
+4.82 points with 1.5B and 7B models, respectively. Additionally,
CurES exhibits faster convergence compared to baselines, including GRPO.
68de25726024653e8a3ed1f8
Yongcheng Zeng
68de25726024653e8a3ed1f9
Zexu Sun
claimed_verified
2025-10-02T13:54:41.772Z
65d1b42f3da87ce21e33261a
/avatars/041cb441fa3871acde4ba565632056bf.svg
RubinSun
user
RubinSun
68de25726024653e8a3ed1fa
Bokai Ji
68de25726024653e8a3ed1fb
Erxue Min
68de25726024653e8a3ed1fc
Hengyi Cai
68de25726024653e8a3ed1fd
Shuaiqiang Wang
68de25726024653e8a3ed1fe
Dawei Yin
68de25726024653e8a3ed1ff
Haifeng Zhang
68de25726024653e8a3ed200
Xu Chen
68de25726024653e8a3ed201
Jun Wang
CurES: From Gradient Analysis to Efficient Curriculum Learning for
  Reasoning LLMs
Curriculum learning plays a crucial role in enhancing the training efficiency
of large language models (LLMs) on reasoning tasks. However, existing methods
often fail to adequately account for variations in prompt difficulty or rely on
simplistic filtering mechanisms to select prompt datasets within a narrow
criterion range, resulting in significant computational waste. In this work, we
approach the problem from the perspective of reinforcement learning gradient
optimization, offering a systematic and theoretical investigation into how to
improve the training efficiency of LLMs. We identify two key factors
influencing training efficiency: the selection of training prompts and the
allocation of rollout quantities across different prompts. Our theoretical
analysis reveals that the sampling distribution of prompts dictates the
convergence rate of gradient descent, while the allocation of the rollout
quantity influences the consistency and stability of overall gradient updates.
Based on these insights, we propose CurES, an efficient training method that
accelerates convergence and employs Bayesian posterior estimation to minimize
computational overhead. Experiments demonstrate that our CurES outperforms
Group Relative Policy Optimization (GRPO) by +3.30 points and
+4.82 points with 1.5B and 7B models, respectively. Additionally,
CurES exhibits faster convergence compared to baselines, including GRPO.
CurES, a reinforcement learning-based method, improves the training efficiency of large language models by optimizing prompt selection and rollout allocation, leading to faster convergence and reduced computational overhead.
curriculum learning large language models reinforcement learning gradient optimization prompt selection rollout quantities gradient descent Bayesian posterior estimation Group Relative Policy Optimization