Pass@k Training for Adaptively Balancing Exploration and Exploitation of
  Large Reasoning Models
Reinforcement learning with verifiable rewards (RLVR), which typically adopts
Pass@1 as the reward, has faced the issues in balancing exploration and
exploitation, causing policies to prefer conservative actions, converging to a
local optimum. Identifying an appropriate reward metric is therefore crucial.
Regarding the prior work, although Pass@k has been used in evaluation, its
connection to LLM exploration ability in RLVR remains largely overlooked. To
investigate this, we first use Pass@k as the reward to train the policy model
(i.e., Pass@k Training), and observe the improvement on its
exploration ability. Next, we derive an analytical solution for the advantage
of Pass@k Training, leading to an efficient and effective process. Building on
this, our analysis reveals that exploration and exploitation are not inherently
conflicting objectives, while they can mutually enhance each other. Moreover,
Pass@k Training with analytical derivation essentially involves directly
designing the advantage function. Inspired by this, we preliminarily explore
the advantage design for RLVR, showing promising results and highlighting a
potential future direction.
689ec27fa4caabb4320e5df4
Zhipeng Chen
claimed_verified
2025-08-18T06:57:05.338Z
629b765ce1af194c641fcbc6
/avatars/7c53a4c2a1e528c19641a2b601731754.svg
Zhipeng Chen
user
TimothyCzp
689ec27fa4caabb4320e5df5
Xiaobo Qin
689ec27fa4caabb4320e5df6
Youbin Wu
689ec27fa4caabb4320e5df7
Yue Ling
689ec27fa4caabb4320e5df8
Qinghao Ye
689ec27fa4caabb4320e5df9
Wayne Xin Zhao
689ec27fa4caabb4320e5dfa
Guang Shi
Pass@k Training for Adaptively Balancing Exploration and Exploitation of
  Large Reasoning Models
Reinforcement learning with verifiable rewards (RLVR), which typically adopts
Pass@1 as the reward, has faced the issues in balancing exploration and
exploitation, causing policies to prefer conservative actions, converging to a
local optimum. Identifying an appropriate reward metric is therefore crucial.
Regarding the prior work, although Pass@k has been used in evaluation, its
connection to LLM exploration ability in RLVR remains largely overlooked. To
investigate this, we first use Pass@k as the reward to train the policy model
(i.e., Pass@k Training), and observe the improvement on its
exploration ability. Next, we derive an analytical solution for the advantage
of Pass@k Training, leading to an efficient and effective process. Building on
this, our analysis reveals that exploration and exploitation are not inherently
conflicting objectives, while they can mutually enhance each other. Moreover,
Pass@k Training with analytical derivation essentially involves directly
designing the advantage function. Inspired by this, we preliminarily explore
the advantage design for RLVR, showing promising results and highlighting a
potential future direction.
Using Pass@k as a reward in reinforcement learning with verifiable rewards improves exploration and reveals that exploration and exploitation can mutually enhance each other.
reinforcement learning verifiable rewards RLVR Pass@1 Pass@k policy model exploration exploitation local optimum advantage function