EPO: Entropy-regularized Policy Optimization for LLM Agents
  Reinforcement Learning
Training LLM agents in multi-turn environments with sparse rewards, where
completing a single task requires 30+ turns of interaction within an episode,
presents a fundamental challenge for reinforcement learning. We identify a
critical failure mode unique to this setting: the exploration-exploitation
cascade failure. This cascade begins with early-stage policy premature
convergence, where sparse feedback causes agents to commit to flawed,
low-entropy strategies. Subsequently, agents enter late-stage policy collapse,
where conventional entropy regularization becomes counterproductive, promoting
chaotic exploration that destabilizes training. We propose Entropy-regularized
Policy Optimization (EPO), a general framework that breaks this failure cycle
through three synergistic mechanisms: (1) adopting entropy regularization in
multi-turn settings to enhance exploration, (2) an entropy smoothing
regularizer that bounds policy entropy within historical averages to prevent
abrupt fluctuations, and (3) adaptive phase-based weighting that balances
exploration and exploitation across training. Our analysis justifies that EPO
guarantees monotonically decreasing entropy variance while maintaining
convergence. EPO achieves up to 152% performance improvement on ScienceWorld
and up to 19.8% on ALFWorld. Our work demonstrates that multi-turn
sparse-reward settings require fundamentally different entropy control than
traditional RL, with broad implications for LLM agent training.
68d9e47c0177a6054b013a13
Xu Wujiang
claimed_verified
2025-09-29T05:59:06.613Z
678523aee670c62966974feb
/avatars/0a0619232f170e517e05657e0ca84699.svg
Wujiang Xu
user
Iscarrot
68d9e47c0177a6054b013a14
Wentian Zhao
68d9e47c0177a6054b013a15
Zhenting Wang
claimed_verified
2025-09-29T11:27:35.286Z
64dfcc62e8b6f3f3baa950e0
/avatars/21bbff67d46c08044efe2406575aa77e.svg
Zhenting Wang
user
ztwang
68d9e47c0177a6054b013a16
Li Yu-Jhe
68d9e47c0177a6054b013a17
Jin Can
68d9e47c0177a6054b013a18
Jin Mingyu
claimed_verified
2025-10-01T10:25:12.569Z
68db3ff5153d1470c35ed4fb
/avatars/11cd11b2ac430ade73e13567de387065.svg
Mingyu Jin
user
JimmyNLP
68d9e47c0177a6054b013a19
Mei Kai
claimed_verified
2025-10-01T15:29:37.623Z
63ea62855c837d9968eb93f7
/avatars/405ea6b595c36251d61cb5d7b7ec87f1.svg
KaiMei
user
dongyuanjushi
68d9e47c0177a6054b013a1a
Wan Kun
claimed_verified
2025-09-29T05:59:02.080Z
66274e02348a5304435dc9cc
/avatars/bda87559cd497c310597c2fc8430b31f.svg
Kun Wan
user
timecuriosity
68d9e47c0177a6054b013a1b
Metaxas Dimitris
EPO: Entropy-regularized Policy Optimization for LLM Agents
  Reinforcement Learning
Training LLM agents in multi-turn environments with sparse rewards, where
completing a single task requires 30+ turns of interaction within an episode,
presents a fundamental challenge for reinforcement learning. We identify a
critical failure mode unique to this setting: the exploration-exploitation
cascade failure. This cascade begins with early-stage policy premature
convergence, where sparse feedback causes agents to commit to flawed,
low-entropy strategies. Subsequently, agents enter late-stage policy collapse,
where conventional entropy regularization becomes counterproductive, promoting
chaotic exploration that destabilizes training. We propose Entropy-regularized
Policy Optimization (EPO), a general framework that breaks this failure cycle
through three synergistic mechanisms: (1) adopting entropy regularization in
multi-turn settings to enhance exploration, (2) an entropy smoothing
regularizer that bounds policy entropy within historical averages to prevent
abrupt fluctuations, and (3) adaptive phase-based weighting that balances
exploration and exploitation across training. Our analysis justifies that EPO
guarantees monotonically decreasing entropy variance while maintaining
convergence. EPO achieves up to 152% performance improvement on ScienceWorld
and up to 19.8% on ALFWorld. Our work demonstrates that multi-turn
sparse-reward settings require fundamentally different entropy control than
traditional RL, with broad implications for LLM agent training.
Entropy-regularized Policy Optimization (EPO) addresses exploration-exploitation challenges in multi-turn environments with sparse rewards, improving performance in tasks like ScienceWorld and ALFWorld.
reinforcement learning exploration-exploitation cascade failure premature convergence policy collapse entropy regularization entropy smoothing regularizer adaptive phase-based weighting entropy variance ScienceWorld ALFWorld