The Choice of Divergence: A Neglected Key to Mitigating Diversity
  Collapse in Reinforcement Learning with Verifiable Reward
A central paradox in fine-tuning Large Language Models (LLMs) with
Reinforcement Learning with Verifiable Reward (RLVR) is the frequent
degradation of multi-attempt performance (Pass@k) despite improvements in
single-attempt accuracy (Pass@1). This is often accompanied by catastrophic
forgetting, where models lose previously acquired skills. While various methods
have been proposed, the choice and function of the divergence term have been
surprisingly unexamined as a proactive solution. We argue that standard RLVR
objectives -- both those using the mode-seeking reverse KL-divergence and those
forgoing a divergence term entirely -- lack a crucial mechanism for knowledge
retention. The reverse-KL actively accelerates this decay by narrowing the
policy, while its absence provides no safeguard against the model drifting from
its diverse knowledge base. We propose a fundamental shift in perspective:
using the divergence term itself as the solution. Our framework,
Diversity-Preserving Hybrid RL (DPH-RL), leverages mass-covering f-divergences
(like forward-KL and JS-divergence) to function as a rehearsal mechanism. By
continuously referencing the initial policy, this approach forces the model to
maintain broad solution coverage. Extensive experiments on math and SQL
generation demonstrate that DPH-RL not only resolves the Pass@k degradation but
improves both Pass@1 and Pass@k in- and out-of-domain. Additionally, DPH-RL is
more training-efficient because it computes f-divergence using generator
functions, requiring only sampling from the initial policy and no online
reference model. Our work highlights a crucial, overlooked axis for improving
RLVR, demonstrating that the proper selection of a divergence measure is a
powerful tool for building more general and diverse reasoning models.
68c2652d29b8ec9932cd09f6
Long Li
68c2652d29b8ec9932cd09f7
Jiaran Hao
68c2652d29b8ec9932cd09f8
Jason Klein Liu
68c2652d29b8ec9932cd09f9
Zhijian Zhou
68c2652d29b8ec9932cd09fa
Xiaoyu Tan
68c2652d29b8ec9932cd09fb
Wei Chu
68c2652d29b8ec9932cd09fc
Zhe Wang
68c2652d29b8ec9932cd09fd
Shirui Pan
68c2652d29b8ec9932cd09fe
Chao Qu
68c2652d29b8ec9932cd09ff
Yuan Qi
The Choice of Divergence: A Neglected Key to Mitigating Diversity
  Collapse in Reinforcement Learning with Verifiable Reward
A central paradox in fine-tuning Large Language Models (LLMs) with
Reinforcement Learning with Verifiable Reward (RLVR) is the frequent
degradation of multi-attempt performance (Pass@k) despite improvements in
single-attempt accuracy (Pass@1). This is often accompanied by catastrophic
forgetting, where models lose previously acquired skills. While various methods
have been proposed, the choice and function of the divergence term have been
surprisingly unexamined as a proactive solution. We argue that standard RLVR
objectives -- both those using the mode-seeking reverse KL-divergence and those
forgoing a divergence term entirely -- lack a crucial mechanism for knowledge
retention. The reverse-KL actively accelerates this decay by narrowing the
policy, while its absence provides no safeguard against the model drifting from
its diverse knowledge base. We propose a fundamental shift in perspective:
using the divergence term itself as the solution. Our framework,
Diversity-Preserving Hybrid RL (DPH-RL), leverages mass-covering f-divergences
(like forward-KL and JS-divergence) to function as a rehearsal mechanism. By
continuously referencing the initial policy, this approach forces the model to
maintain broad solution coverage. Extensive experiments on math and SQL
generation demonstrate that DPH-RL not only resolves the Pass@k degradation but
improves both Pass@1 and Pass@k in- and out-of-domain. Additionally, DPH-RL is
more training-efficient because it computes f-divergence using generator
functions, requiring only sampling from the initial policy and no online
reference model. Our work highlights a crucial, overlooked axis for improving
RLVR, demonstrating that the proper selection of a divergence measure is a
powerful tool for building more general and diverse reasoning models.
A new framework, DPH-RL, uses mass-covering f-divergences to address Pass@k degradation and catastrophic forgetting in fine-tuning LLMs with RLVR, improving both Pass@1 and Pass@k.
Large Language Models Reinforcement Learning with Verifiable Reward RLVR multi-attempt performance Pass@k single-attempt accuracy Pass@1 catastrophic forgetting reverse KL-divergence divergence term knowledge retention mass-covering f-divergences forward-KL JS-divergence rehearsal mechanism initial policy solution coverage training-efficiency generator functions online reference model