Information Gain-based Policy Optimization: A Simple and Effective
  Approach for Multi-Turn LLM Agents
Large language model (LLM)-based agents are increasingly trained with
reinforcement learning (RL) to enhance their ability to interact with external
environments through tool use, particularly in search-based settings that
require multi-turn reasoning and knowledge acquisition. However, existing
approaches typically rely on outcome-based rewards that are only provided at
the final answer. This reward sparsity becomes particularly problematic in
multi-turn settings, where long trajectories exacerbate two critical issues:
(i) advantage collapse, where all rollouts receive identical rewards and
provide no useful learning signals, and (ii) lack of fine-grained credit
assignment, where dependencies between turns are obscured, especially in
long-horizon tasks. In this paper, we propose Information Gain-based Policy
Optimization (IGPO), a simple yet effective RL framework that provides dense
and intrinsic supervision for multi-turn agent training. IGPO models each
interaction turn as an incremental process of acquiring information about the
ground truth, and defines turn-level rewards as the marginal increase in the
policy's probability of producing the correct answer. Unlike prior
process-level reward approaches that depend on external reward models or costly
Monte Carlo estimation, IGPO derives intrinsic rewards directly from the
model's own belief updates. These intrinsic turn-level rewards are combined
with outcome-level supervision to form dense reward trajectories. Extensive
experiments on both in-domain and out-of-domain benchmarks demonstrate that
IGPO consistently outperforms strong baselines in multi-turn scenarios,
achieving higher accuracy and improved sample efficiency.
68f1a04a6e0bef323a68fcac
Guoqing Wang
68f1a04a6e0bef323a68fcad
Sunhao Dai
68f1a04a6e0bef323a68fcae
Guangze Ye
68f1a04a6e0bef323a68fcaf
Zeyu Gan
68f1a04a6e0bef323a68fcb0
Wei Yao
68f1a04a6e0bef323a68fcb1
Yong Deng
68f1a04a6e0bef323a68fcb2
Xiaofeng Wu
68f1a04a6e0bef323a68fcb3
Zhenzhe Ying
Information Gain-based Policy Optimization: A Simple and Effective
  Approach for Multi-Turn LLM Agents
Large language model (LLM)-based agents are increasingly trained with
reinforcement learning (RL) to enhance their ability to interact with external
environments through tool use, particularly in search-based settings that
require multi-turn reasoning and knowledge acquisition. However, existing
approaches typically rely on outcome-based rewards that are only provided at
the final answer. This reward sparsity becomes particularly problematic in
multi-turn settings, where long trajectories exacerbate two critical issues:
(i) advantage collapse, where all rollouts receive identical rewards and
provide no useful learning signals, and (ii) lack of fine-grained credit
assignment, where dependencies between turns are obscured, especially in
long-horizon tasks. In this paper, we propose Information Gain-based Policy
Optimization (IGPO), a simple yet effective RL framework that provides dense
and intrinsic supervision for multi-turn agent training. IGPO models each
interaction turn as an incremental process of acquiring information about the
ground truth, and defines turn-level rewards as the marginal increase in the
policy's probability of producing the correct answer. Unlike prior
process-level reward approaches that depend on external reward models or costly
Monte Carlo estimation, IGPO derives intrinsic rewards directly from the
model's own belief updates. These intrinsic turn-level rewards are combined
with outcome-level supervision to form dense reward trajectories. Extensive
experiments on both in-domain and out-of-domain benchmarks demonstrate that
IGPO consistently outperforms strong baselines in multi-turn scenarios,
achieving higher accuracy and improved sample efficiency.
Information Gain-based Policy Optimization (IGPO) enhances multi-turn reasoning in large language models by providing dense intrinsic rewards derived from the model's belief updates, improving accuracy and sample efficiency.
reinforcement learning large language model tool use multi-turn reasoning knowledge acquisition reward sparsity advantage collapse fine-grained credit assignment Information Gain-based Policy Optimization intrinsic rewards belief updates dense reward trajectories sample efficiency