Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains
  RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a key paradigm for post-training Large Language Models (LLMs), particularly for
complex reasoning tasks. However, vanilla RLVR training has been shown to
improve Pass@1 performance at the expense of policy entropy, leading to reduced
generation diversity and limiting the Pass@k performance, which typically
represents the upper bound of LLM reasoning capability. In this paper, we
systematically analyze the policy's generation diversity from the perspective
of training problems and find that augmenting and updating training problems
helps mitigate entropy collapse during training. Based on these observations,
we propose an online Self-play with Variational problem Synthesis (SvS)
strategy for RLVR training, which uses the policy's correct solutions to
synthesize variational problems while ensuring their reference answers remain
identical to the originals. This self-improving strategy effectively maintains
policy entropy during training and substantially improves Pass@k compared with
standard RLVR, sustaining prolonged improvements and achieving absolute gains
of 18.3% and 22.8% in Pass@32 performance on the competition-level AIME24 and
AIME25 benchmarks. Experiments on 12 reasoning benchmarks across varying model
sizes from 3B to 32B consistently demonstrate the generalizability and
robustness of SvS.
68a93fc286b21a0e2e358862
Xiao Liang
claimed_verified
2025-08-25T08:20:26.427Z
6560763e152b659e623865ae
https://cdn-avatars.huggingface.co/v1/production/uploads/6560763e152b659e623865ae/cTT2jGnPU_8XMrUTvqZ2h.jpeg
Xiao Liang
user
MasterVito
68a93fc286b21a0e2e358863
Zhongzhi Li
68a93fc286b21a0e2e358864
Yeyun Gong
68a93fc286b21a0e2e358865
Yelong Shen
68a93fc286b21a0e2e358866
Ying Nian Wu
68a93fc286b21a0e2e358867
Zhijiang Guo
68a93fc286b21a0e2e358868
Weizhu Chen
Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains
  RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as
a key paradigm for post-training Large Language Models (LLMs), particularly for
complex reasoning tasks. However, vanilla RLVR training has been shown to
improve Pass@1 performance at the expense of policy entropy, leading to reduced
generation diversity and limiting the Pass@k performance, which typically
represents the upper bound of LLM reasoning capability. In this paper, we
systematically analyze the policy's generation diversity from the perspective
of training problems and find that augmenting and updating training problems
helps mitigate entropy collapse during training. Based on these observations,
we propose an online Self-play with Variational problem Synthesis (SvS)
strategy for RLVR training, which uses the policy's correct solutions to
synthesize variational problems while ensuring their reference answers remain
identical to the originals. This self-improving strategy effectively maintains
policy entropy during training and substantially improves Pass@k compared with
standard RLVR, sustaining prolonged improvements and achieving absolute gains
of 18.3% and 22.8% in Pass@32 performance on the competition-level AIME24 and
AIME25 benchmarks. Experiments on 12 reasoning benchmarks across varying model
sizes from 3B to 32B consistently demonstrate the generalizability and
robustness of SvS.
An online self-play strategy with variational problem synthesis for RLVR training maintains policy entropy and improves Pass@k performance on reasoning benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) Large Language Models (LLMs) policy entropy generation diversity Pass@1 Pass@k self-play variational problem synthesis AIME24 AIME25