Agentic Entropy-Balanced Policy Optimization
Recently, Agentic Reinforcement Learning (Agentic RL) has made significant
progress in incentivizing the multi-turn, long-horizon tool-use capabilities of
web agents. While mainstream agentic RL algorithms autonomously explore
high-uncertainty tool-call steps under the guidance of entropy, excessive
reliance on entropy signals can impose further constraints, leading to the
training collapse. In this paper, we delve into the challenges caused by
entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an
agentic RL algorithm designed to balance entropy in both the rollout and policy
update phases. AEPO comprises two core components: (1) a dynamic
entropy-balanced rollout mechanism that adaptively allocate global and branch
sampling budget through entropy pre-monitoring, while imposing a branch penalty
on consecutive high-entropy tool-call steps to prevent over-branching issues;
and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient
operation into the high-entropy clipping term to preserve and properly rescale
gradients on high-entropy tokens, while incorporating entropy-aware advantage
estimation to prioritize learning on high-uncertainty tokens. Results across 14
challenging datasets show that AEPO consistently outperforms 7 mainstream RL
algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive
results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker
for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on
WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout
sampling diversity while maintaining stable policy entropy, facilitating
scalable web agent training.
68f1a44e6e0bef323a68fd02
Guanting Dong
claimed_verified
2025-10-17T04:10:43.546Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
68f1a44e6e0bef323a68fd03
Licheng Bao
claimed_verified
2025-10-17T04:10:38.484Z
6690cf4c5d0a25eea5e3dfbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6690cf4c5d0a25eea5e3dfbc/zqY3kl0uZaXrOPJvfmRAn.jpeg
baolicheng
user
blc0910
68f1a44e6e0bef323a68fd04
Zhongyuan Wang
68f1a44e6e0bef323a68fd05
Kangzhi Zhao
68f1a44e6e0bef323a68fd06
Xiaoxi Li
68f1a44e6e0bef323a68fd07
Jiajie Jin
68f1a44e6e0bef323a68fd08
Jinghan Yang
68f1a44e6e0bef323a68fd09
Hangyu Mao
68f1a44e6e0bef323a68fd0a
Fuzheng Zhang
68f1a44e6e0bef323a68fd0b
Kun Gai
68f1a44e6e0bef323a68fd0c
Guorui Zhou
68f1a44e6e0bef323a68fd0d
Yutao Zhu
68f1a44e6e0bef323a68fd0e
Ji-Rong Wen
68f1a44e6e0bef323a68fd0f
Zhicheng Dou
Agentic Entropy-Balanced Policy Optimization
Recently, Agentic Reinforcement Learning (Agentic RL) has made significant
progress in incentivizing the multi-turn, long-horizon tool-use capabilities of
web agents. While mainstream agentic RL algorithms autonomously explore
high-uncertainty tool-call steps under the guidance of entropy, excessive
reliance on entropy signals can impose further constraints, leading to the
training collapse. In this paper, we delve into the challenges caused by
entropy and propose the Agentic Entropy-Balanced Policy Optimization (AEPO), an
agentic RL algorithm designed to balance entropy in both the rollout and policy
update phases. AEPO comprises two core components: (1) a dynamic
entropy-balanced rollout mechanism that adaptively allocate global and branch
sampling budget through entropy pre-monitoring, while imposing a branch penalty
on consecutive high-entropy tool-call steps to prevent over-branching issues;
and (2) Entropy-Balanced Policy Optimization that inserts a stop-gradient
operation into the high-entropy clipping term to preserve and properly rescale
gradients on high-entropy tokens, while incorporating entropy-aware advantage
estimation to prioritize learning on high-uncertainty tokens. Results across 14
challenging datasets show that AEPO consistently outperforms 7 mainstream RL
algorithms. With just 1K RL samples, Qwen3-14B with AEPO achieves impressive
results: 47.6% on GAIA, 11.2% on Humanity's Last Exam, and 43.0% on WebWalker
for Pass@1; 65.0% on GAIA, 26.0% on Humanity's Last Exam, and 70.0% on
WebWalker for Pass@5. Further analysis reveals that AEPO improves rollout
sampling diversity while maintaining stable policy entropy, facilitating
scalable web agent training.
AEPO, an agentic RL algorithm, addresses entropy-related challenges in web agent training, enhancing performance and stability across various datasets.
Agentic Reinforcement Learning AEPO entropy rollout policy update dynamic entropy-balanced rollout branch penalty stop-gradient operation high-entropy clipping entropy-aware advantage estimation rollout sampling diversity policy entropy