Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement
  Learning
Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm
for enhancing the reasoning capabilities of large language models (LLMs), yet
its success hinges on effective exploration. An ideal exploration strategy must
navigate two fundamental challenges: it must preserve sample quality while also
ensuring training stability. While standard fixed-temperature sampling is
simple, it struggles to balance these competing demands, as high temperatures
degrade sample quality and low temperatures limit discovery. In this work, we
propose a simpler and more effective strategy, Exploratory Annealed Decoding
(EAD), grounded in the insight that exploration is most impactful on early
tokens which define a sequence's semantic direction. EAD implements an
intuitive **explore-at-the-beginning, exploit-at-the-end** strategy by
annealing the sampling temperature from high to low during generation. This
dynamic schedule encourages meaningful, high-level diversity at the start, then
gradually lowers the temperature to preserve sample quality and keep the
sampling distribution close to the target policy, which is essential for stable
training. We demonstrate that EAD is a lightweight, plug-and-play method that
significantly improves sample efficiency, consistently outperforming
fixed-temperature sampling across various RLVR algorithms and model sizes. Our
work suggests that aligning exploration with the natural dynamics of sequential
generation offers a robust path to improving LLM reasoning.
68e692c60d37f65d7721a604
Chenghao Yang
68e692c60d37f65d7721a605
Lin Gui
68e692c60d37f65d7721a606
Chenxiao Yang
68e692c60d37f65d7721a607
Victor Veitch
68e692c60d37f65d7721a608
Lizhu Zhang
68e692c60d37f65d7721a609
Zhuokai Zhao
Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement
  Learning
Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm
for enhancing the reasoning capabilities of large language models (LLMs), yet
its success hinges on effective exploration. An ideal exploration strategy must
navigate two fundamental challenges: it must preserve sample quality while also
ensuring training stability. While standard fixed-temperature sampling is
simple, it struggles to balance these competing demands, as high temperatures
degrade sample quality and low temperatures limit discovery. In this work, we
propose a simpler and more effective strategy, Exploratory Annealed Decoding
(EAD), grounded in the insight that exploration is most impactful on early
tokens which define a sequence's semantic direction. EAD implements an
intuitive **explore-at-the-beginning, exploit-at-the-end** strategy by
annealing the sampling temperature from high to low during generation. This
dynamic schedule encourages meaningful, high-level diversity at the start, then
gradually lowers the temperature to preserve sample quality and keep the
sampling distribution close to the target policy, which is essential for stable
training. We demonstrate that EAD is a lightweight, plug-and-play method that
significantly improves sample efficiency, consistently outperforming
fixed-temperature sampling across various RLVR algorithms and model sizes. Our
work suggests that aligning exploration with the natural dynamics of sequential
generation offers a robust path to improving LLM reasoning.
Exploratory Annealed Decoding (EAD) improves sample efficiency in reinforcement learning with verifiable rewards by dynamically adjusting the sampling temperature during generation.
reinforcement learning with verifiable rewards large language models exploration strategy sample quality training stability fixed-temperature sampling exploratory annealed decoding explore-at-the-beginning exploit-at-the-end sampling temperature target policy sample efficiency sequential generation