VCRL: Variance-based Curriculum Reinforcement Learning for Large
  Language Models
Policy-based reinforcement learning currently plays an important role in
improving LLMs on mathematical reasoning tasks. However, existing rollout-based
reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly
consider LLMs' learning ability for samples of different difficulty levels,
which is contrary to the human cognitive process of mathematical reasoning
tasks from easy to difficult. Intuitively, we find that the variance of the
rollout group's reward in RLVR partly reflects the difficulty of the current
sample for LLMs. Samples that are too easy or too difficult have a lower
variance, while samples with moderate difficulty have a higher variance. Based
on this, we propose VCRL, a curriculum reinforcement learning framework that
dynamically controls the difficulty of training samples based on the variance
of group rewards. Experiments on five mathematical benchmarks and two models
reveal the advantages of VCRL over the current LLM RL baselines.
68d607ad8ccd91bdd39ffe04
Guochao Jiang
claimed_verified
2025-09-26T12:16:44.222Z
644a1dbb9c340e5e1e713153
/avatars/21cb93ad067a798a39829ef7e67c70b8.svg
JGC
user
Nothing2Say
68d607ad8ccd91bdd39ffe05
Wenfeng Feng
claimed_verified
2025-09-26T15:35:56.925Z
68b691341c52b8b741ba7fd4
/avatars/efae3c632577565eb3645e64d4f725ac.svg
wenfeng feng
user
wenfengfwf
68d607ad8ccd91bdd39ffe06
Guofeng Quan
68d607ad8ccd91bdd39ffe07
Chuzhan Hao
claimed_verified
2025-09-26T12:16:38.565Z
64ae631b58bd9e9cc2f5a749
/avatars/ce6426ec3bdb618a9e449297e7f147e0.svg
Chuzhan HAO
user
Chuzhan
68d607ad8ccd91bdd39ffe08
Yuewei Zhang
claimed_verified
2025-09-26T15:35:58.643Z
68b6ec9b1224e891a08873bd
/avatars/58604eab370fe35ee1053a1c33874982.svg
yuewei zhang
user
zhangywlfh
68d607ad8ccd91bdd39ffe09
Guohua Liu
68d607ad8ccd91bdd39ffe0a
Hao Wang
VCRL: Variance-based Curriculum Reinforcement Learning for Large
  Language Models
Policy-based reinforcement learning currently plays an important role in
improving LLMs on mathematical reasoning tasks. However, existing rollout-based
reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly
consider LLMs' learning ability for samples of different difficulty levels,
which is contrary to the human cognitive process of mathematical reasoning
tasks from easy to difficult. Intuitively, we find that the variance of the
rollout group's reward in RLVR partly reflects the difficulty of the current
sample for LLMs. Samples that are too easy or too difficult have a lower
variance, while samples with moderate difficulty have a higher variance. Based
on this, we propose VCRL, a curriculum reinforcement learning framework that
dynamically controls the difficulty of training samples based on the variance
of group rewards. Experiments on five mathematical benchmarks and two models
reveal the advantages of VCRL over the current LLM RL baselines.
A curriculum reinforcement learning framework dynamically adjusts training sample difficulty based on reward variance, improving LLM performance on mathematical reasoning tasks.
policy-based reinforcement learning rollout-based reinforcement learning GRPO DAPO GSPO RLVR curriculum reinforcement learning VCRL mathematical reasoning tasks reward variance