ΔL Normalization: Rethink Loss Aggregation in RLVR
We propose Delta L Normalization, a simple yet effective loss aggregation
method tailored to the characteristic of dynamic generation lengths in
Reinforcement Learning with Verifiable Rewards (RLVR). Recently, RLVR has
demonstrated strong potential in improving the reasoning capabilities of large
language models (LLMs), but a major challenge lies in the large variability of
response lengths during training, which leads to high gradient variance and
unstable optimization. Although previous methods such as GRPO, DAPO, and Dr.
GRPO introduce different loss normalization terms to address this issue, they
either produce biased estimates or still suffer from high gradient variance. By
analyzing the effect of varying lengths on policy loss both theoretically and
empirically, we reformulate the problem as finding a minimum-variance unbiased
estimator. Our proposed Delta L Normalization not only provides an unbiased
estimate of the true policy loss but also minimizes gradient variance in
theory. Extensive experiments show that it consistently achieves superior
results across different model sizes, maximum lengths, and tasks. Our code will
be made public at https://github.com/zerolllin/Delta-L-Normalization.
68c132223912ed54cf5432f8
Zhiyuan He
68c132223912ed54cf5432f9
Xufang Luo
68c132223912ed54cf5432fa
Yike Zhang
68c132223912ed54cf5432fb
Yuqing Yang
68c132223912ed54cf5432fc
Lili Qiu
ΔL Normalization: Rethink Loss Aggregation in RLVR
We propose Delta L Normalization, a simple yet effective loss aggregation
method tailored to the characteristic of dynamic generation lengths in
Reinforcement Learning with Verifiable Rewards (RLVR). Recently, RLVR has
demonstrated strong potential in improving the reasoning capabilities of large
language models (LLMs), but a major challenge lies in the large variability of
response lengths during training, which leads to high gradient variance and
unstable optimization. Although previous methods such as GRPO, DAPO, and Dr.
GRPO introduce different loss normalization terms to address this issue, they
either produce biased estimates or still suffer from high gradient variance. By
analyzing the effect of varying lengths on policy loss both theoretically and
empirically, we reformulate the problem as finding a minimum-variance unbiased
estimator. Our proposed Delta L Normalization not only provides an unbiased
estimate of the true policy loss but also minimizes gradient variance in
theory. Extensive experiments show that it consistently achieves superior
results across different model sizes, maximum lengths, and tasks. Our code will
be made public at https://github.com/zerolllin/Delta-L-Normalization.
ΔL Normalization addresses gradient variance in Reinforcement Learning with Verifiable Rewards by providing an unbiased policy loss estimate with minimal variance.
ΔL Normalization loss aggregation dynamic generation lengths Reinforcement Learning with Verifiable Rewards RLVR large language models LLMs gradient variance policy loss unbiased estimate GRPO DAPO Dr. GRPO