Agentic Reinforced Policy Optimization
Large-scale reinforcement learning with verifiable rewards (RLVR) has
demonstrated its effectiveness in harnessing the potential of large language
models (LLMs) for single-turn reasoning tasks. In realistic reasoning
scenarios, LLMs can often utilize external tools to assist in task-solving
processes. However, current RL algorithms inadequately balance the models'
intrinsic long-horizon reasoning capabilities and their proficiency in
multi-turn tool interactions. To bridge this gap, we propose Agentic Reinforced
Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training
multi-turn LLM-based agents. Through preliminary experiments, we observe that
LLMs tend to exhibit highly uncertain behavior, characterized by an increase in
the entropy distribution of generated tokens, immediately following
interactions with external tools. Motivated by this observation, ARPO
incorporates an entropy-based adaptive rollout mechanism, dynamically balancing
global trajectory sampling and step-level sampling, thereby promoting
exploration at steps with high uncertainty after tool usage. By integrating an
advantage attribution estimation, ARPO enables LLMs to internalize advantage
differences in stepwise tool-use interactions. Our experiments across 13
challenging benchmarks in computational reasoning, knowledge reasoning, and
deep search domains demonstrate ARPO's superiority over trajectory-level RL
algorithms. Remarkably, ARPO achieves improved performance using only half of
the tool-use budget required by existing methods, offering a scalable solution
for aligning LLM-based agents with real-time dynamic environments. Our code and
datasets are released at https://github.com/dongguanting/ARPO
688836e8af872d625c10c60d
Guanting Dong
claimed_verified
2025-07-29T12:51:05.404Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
688836e8af872d625c10c60e
Hangyu Mao
688836e8af872d625c10c60f
Kai Ma
688836e8af872d625c10c610
Licheng Bao
688836e8af872d625c10c611
Yifei Chen
688836e8af872d625c10c612
Zhongyuan Wang
688836e8af872d625c10c613
Zhongxia Chen
688836e8af872d625c10c614
Jiazhen Du
688836e8af872d625c10c615
Huiyang Wang
688836e8af872d625c10c616
Fuzheng Zhang
688836e8af872d625c10c617
Guorui Zhou
688836e8af872d625c10c618
Yutao Zhu
688836e8af872d625c10c619
Ji-Rong Wen
688836e8af872d625c10c61a
Zhicheng Dou
Agentic Reinforced Policy Optimization
Large-scale reinforcement learning with verifiable rewards (RLVR) has
demonstrated its effectiveness in harnessing the potential of large language
models (LLMs) for single-turn reasoning tasks. In realistic reasoning
scenarios, LLMs can often utilize external tools to assist in task-solving
processes. However, current RL algorithms inadequately balance the models'
intrinsic long-horizon reasoning capabilities and their proficiency in
multi-turn tool interactions. To bridge this gap, we propose Agentic Reinforced
Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training
multi-turn LLM-based agents. Through preliminary experiments, we observe that
LLMs tend to exhibit highly uncertain behavior, characterized by an increase in
the entropy distribution of generated tokens, immediately following
interactions with external tools. Motivated by this observation, ARPO
incorporates an entropy-based adaptive rollout mechanism, dynamically balancing
global trajectory sampling and step-level sampling, thereby promoting
exploration at steps with high uncertainty after tool usage. By integrating an
advantage attribution estimation, ARPO enables LLMs to internalize advantage
differences in stepwise tool-use interactions. Our experiments across 13
challenging benchmarks in computational reasoning, knowledge reasoning, and
deep search domains demonstrate ARPO's superiority over trajectory-level RL
algorithms. Remarkably, ARPO achieves improved performance using only half of
the tool-use budget required by existing methods, offering a scalable solution
for aligning LLM-based agents with real-time dynamic environments. Our code and
datasets are released at https://github.com/dongguanting/ARPO
Agentic Reinforced Policy Optimization (ARPO) enhances multi-turn reasoning in large language models by balancing long-horizon capabilities and tool interactions, using entropy-based adaptive rollouts and advantage attribution.
reinforcement learning verifiable rewards large language models single-turn reasoning multi-turn reasoning external tools Agentic Reinforced Policy Optimization entropy-based adaptive rollout advantage attribution estimation computational reasoning knowledge reasoning deep search domains