SimKO: Simple Pass@K Policy Optimization
Reinforcement learning with verifiable rewards (RLVR) has advanced the
reasoning capabilities of large language models (LLMs). However, prevailing
RLVR methods exhibit a systematic bias toward exploitation over exploration, as
evidenced by improved pass@1 but reduced pass@K (K>1) performance. To
understand this issue, we analyze training dynamics of RLVR methods by tracking
the token-level probability distributions over vocabulary candidates. Our
analysis reveals a consistent probability concentration effect where the top-1
candidate increasingly accumulates probability mass and suppresses that of
other candidates. More importantly, stronger over-concentration correlates with
worse pass@K performance. Inspired by this finding, we propose Simple Pass@K
Optimization (SimKO), a method designed to mitigate the over-concentration
issue, thereby encouraging exploration. SimKO operates in an asymmetrical
manner. For verified-correct responses, it boosts the probabilities of the
top-K candidates. For verified-incorrect responses, it applies stronger
penalties to the top-1 candidate. We observe that this asymmetric design is
particularly effective at mitigating over-concentration when applied at tokens
with high entropy. Across various math and logical-reasoning benchmarks, SimKO
consistently yields higher pass@K for a wide range of K, providing a simple way
to improve RLVR's exploration.
68f1cfb86e0bef323a68ff30
Ruotian Peng
68f1cfb86e0bef323a68ff31
Yi Ren
68f1cfb86e0bef323a68ff32
Zhouliang Yu
claimed_verified
2025-10-18T04:49:15.042Z
62a80fe3ac97233f1625235a
https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
Zhouliang Yu
user
zhouliang
68f1cfb86e0bef323a68ff33
Weiyang Liu
claimed_verified
2025-10-18T04:49:12.561Z
648905d1a15c43c791d4381f
https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
Weiyang Liu
user
wy1iu
68f1cfb86e0bef323a68ff34
Yandong Wen
SimKO: Simple Pass@K Policy Optimization
Reinforcement learning with verifiable rewards (RLVR) has advanced the
reasoning capabilities of large language models (LLMs). However, prevailing
RLVR methods exhibit a systematic bias toward exploitation over exploration, as
evidenced by improved pass@1 but reduced pass@K (K>1) performance. To
understand this issue, we analyze training dynamics of RLVR methods by tracking
the token-level probability distributions over vocabulary candidates. Our
analysis reveals a consistent probability concentration effect where the top-1
candidate increasingly accumulates probability mass and suppresses that of
other candidates. More importantly, stronger over-concentration correlates with
worse pass@K performance. Inspired by this finding, we propose Simple Pass@K
Optimization (SimKO), a method designed to mitigate the over-concentration
issue, thereby encouraging exploration. SimKO operates in an asymmetrical
manner. For verified-correct responses, it boosts the probabilities of the
top-K candidates. For verified-incorrect responses, it applies stronger
penalties to the top-1 candidate. We observe that this asymmetric design is
particularly effective at mitigating over-concentration when applied at tokens
with high entropy. Across various math and logical-reasoning benchmarks, SimKO
consistently yields higher pass@K for a wide range of K, providing a simple way
to improve RLVR's exploration.
Simple Pass@K Optimization (SimKO) addresses over-concentration in reinforcement learning with verifiable rewards (RLVR) by asymmetrically adjusting token probabilities, enhancing exploration and pass@K performance.
reinforcement learning with verifiable rewards RLVR large language models LLMs pass@1 pass@K token-level probability distributions probability concentration effect Simple Pass@K Optimization SimKO entropy