CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning
  in Large Language Models
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm
for enhancing the reasoning ability of Large Language Models (LLMs). Yet
current RLVR methods often explore poorly, leading to premature convergence and
entropy collapse. To address this challenge, we introduce Curiosity-Driven
Exploration (CDE), a framework that leverages the model's own intrinsic sense
of curiosity to guide exploration. We formalize curiosity with signals from
both the actor and the critic: for the actor, we use perplexity over its
generated response, and for the critic, we use the variance of value estimates
from a multi-head architecture. Both signals serve as an exploration bonus
within the RLVR framework to guide the model. Our theoretical analysis shows
that the actor-wise bonus inherently penalizes overconfident errors and
promotes diversity among correct responses; moreover, we connect the
critic-wise bonus to the well-established count-based exploration bonus in RL.
Empirically, our method achieves an approximate +3 point improvement over
standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a
calibration collapse mechanism within RLVR, shedding light on common LLM
failure modes.
68c376fbfc1747b912403987
Runpeng Dai
claimed_verified
2025-09-15T15:08:46.015Z
65037565da2d88e201f63b7a
/avatars/d1b6ce17236360e9583b8bb4cb87e506.svg
Runpeng Dai
user
Leo-Dai
68c376fbfc1747b912403988
Linfeng Song
68c376fbfc1747b912403989
Haolin Liu
68c376fbfc1747b91240398a
Zhenwen Liang
68c376fbfc1747b91240398b
Dian Yu
68c376fbfc1747b91240398c
Haitao Mi
68c376fbfc1747b91240398d
Zhaopeng Tu
68c376fbfc1747b91240398e
Rui Liu
claimed_verified
2025-09-12T16:09:21.335Z
6656bf615b203a05a1f0968c
/avatars/1ee0b0099c10dd76c8e3b7d312221b15.svg
Rui Liu
user
lr10260
68c376fbfc1747b91240398f
Tong Zheng
claimed_verified
2025-09-15T15:08:43.661Z
6623ea65b642e29cdf90a1b4
/avatars/e32e90574c1162b2be87ed78604e3e4d.svg
TongZheng
user
TongZheng1999
68c376fbfc1747b912403990
Hongtu Zhu
68c376fbfc1747b912403991
Dong Yu
CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning
  in Large Language Models
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm
for enhancing the reasoning ability of Large Language Models (LLMs). Yet
current RLVR methods often explore poorly, leading to premature convergence and
entropy collapse. To address this challenge, we introduce Curiosity-Driven
Exploration (CDE), a framework that leverages the model's own intrinsic sense
of curiosity to guide exploration. We formalize curiosity with signals from
both the actor and the critic: for the actor, we use perplexity over its
generated response, and for the critic, we use the variance of value estimates
from a multi-head architecture. Both signals serve as an exploration bonus
within the RLVR framework to guide the model. Our theoretical analysis shows
that the actor-wise bonus inherently penalizes overconfident errors and
promotes diversity among correct responses; moreover, we connect the
critic-wise bonus to the well-established count-based exploration bonus in RL.
Empirically, our method achieves an approximate +3 point improvement over
standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a
calibration collapse mechanism within RLVR, shedding light on common LLM
failure modes.
Curiosity-Driven Exploration (CDE) enhances Reinforcement Learning with Verifiable Rewards (RLVR) by using intrinsic curiosity signals from the actor and critic to improve exploration and reduce premature convergence in Large Language Models (LLMs).
Reinforcement Learning with Verifiable Rewards RLVR Large Language Models LLMs Curiosity-Driven Exploration CDE actor critic perplexity value estimates multi-head architecture exploration bonus overconfident errors diversity count-based exploration bonus calibration collapse