Harnessing Uncertainty: Entropy-Modulated Policy Gradients for
  Long-Horizon LLM Agents
In long-horizon tasks, recent agents based on Large Language Models (LLMs)
face a significant challenge that sparse, outcome-based rewards make it
difficult to assign credit to intermediate steps. Previous methods mainly focus
on creating dense reward signals to guide learning, either through traditional
reinforcement learning techniques like inverse reinforcement learning or by
using Process Reward Models for step-by-step feedback. In this paper, we
identify a fundamental problem in the learning dynamics of LLMs: the magnitude
of policy gradients is inherently coupled with the entropy, which leads to
inefficient small updates for confident correct actions and potentially
destabilizes large updates for uncertain ones. To resolve this, we propose
Entropy-Modulated Policy Gradients (EMPG), a framework that re-calibrates the
learning signal based on step-wise uncertainty and the final task outcome. EMPG
amplifies updates for confident correct actions, penalizes confident errors,
and attenuates updates from uncertain steps to stabilize exploration. We
further introduce a bonus term for future clarity that encourages agents to
find more predictable solution paths. Through comprehensive experiments on
three challenging agent tasks, WebShop, ALFWorld, and Deep Search, we
demonstrate that EMPG achieves substantial performance gains and significantly
outperforms strong policy gradient baselines. Project page is at
https://empgseed-seed.github.io/
68c37f86fc1747b9124039de
Jiawei Wang
claimed_verified
2025-09-12T16:08:25.849Z
64060b49a577649430bf6974
/avatars/74d0d6ed656b593e4c101b09edf18c7a.svg
Jiawei Wang
user
Jarvis1111
68c37f86fc1747b9124039df
Jiacai Liu
68c37f86fc1747b9124039e0
Yuqian Fu
claimed_verified
2025-09-12T16:08:28.961Z
670aa09d35918e99fe7ff6b1
/avatars/5cbea2284165191e96544bacf2bfb50f.svg
Yuqian Fu
user
Yuqian-Fu
68c37f86fc1747b9124039e1
Yingru Li
claimed_verified
2025-09-12T16:08:22.762Z
67277d20eebb94a257cd6925
/avatars/c1f714b59fecb53770e28920c0c267cb.svg
Yingru Li
user
R1ch0rd
68c37f86fc1747b9124039e2
Xintao Wang
68c37f86fc1747b9124039e3
Yuan Lin
68c37f86fc1747b9124039e4
Yu Yue
68c37f86fc1747b9124039e5
Lin Zhang
68c37f86fc1747b9124039e6
Yang Wang
68c37f86fc1747b9124039e7
Ke Wang
Harnessing Uncertainty: Entropy-Modulated Policy Gradients for
  Long-Horizon LLM Agents
In long-horizon tasks, recent agents based on Large Language Models (LLMs)
face a significant challenge that sparse, outcome-based rewards make it
difficult to assign credit to intermediate steps. Previous methods mainly focus
on creating dense reward signals to guide learning, either through traditional
reinforcement learning techniques like inverse reinforcement learning or by
using Process Reward Models for step-by-step feedback. In this paper, we
identify a fundamental problem in the learning dynamics of LLMs: the magnitude
of policy gradients is inherently coupled with the entropy, which leads to
inefficient small updates for confident correct actions and potentially
destabilizes large updates for uncertain ones. To resolve this, we propose
Entropy-Modulated Policy Gradients (EMPG), a framework that re-calibrates the
learning signal based on step-wise uncertainty and the final task outcome. EMPG
amplifies updates for confident correct actions, penalizes confident errors,
and attenuates updates from uncertain steps to stabilize exploration. We
further introduce a bonus term for future clarity that encourages agents to
find more predictable solution paths. Through comprehensive experiments on
three challenging agent tasks, WebShop, ALFWorld, and Deep Search, we
demonstrate that EMPG achieves substantial performance gains and significantly
outperforms strong policy gradient baselines. Project page is at
https://empgseed-seed.github.io/
Entropy-Modulated Policy Gradients (EMPG) addresses learning dynamics issues in LLMs by recalibrating policy gradients based on uncertainty and task outcomes, leading to improved performance in long-horizon tasks.
Large Language Models (LLMs) sparse rewards dense reward signals inverse reinforcement learning Process Reward Models policy gradients entropy Entropy-Modulated Policy Gradients (EMPG) WebShop ALFWorld Deep Search