Multi-Agent Tool-Integrated Policy Optimization
Large language models (LLMs) increasingly rely on multi-turn tool-integrated
planning for knowledge-intensive and complex reasoning tasks. Existing
implementations typically rely on a single agent, but they suffer from limited
context length and noisy tool responses. A natural solution is to adopt a
multi-agent framework with planner- and worker-agents to manage context.
However, no existing methods support effective reinforcement learning
post-training of tool-integrated multi-agent frameworks. To address this gap,
we propose Multi-Agent Tool-Integrated Policy Optimization (MATPO), which
enables distinct roles (planner and worker) to be trained within a single LLM
instance using role-specific prompts via reinforcement learning. MATPO is
derived from a principled credit assignment mechanism across planner and worker
rollouts. This design eliminates the need to deploy multiple LLMs, which would
be memory-intensive, while preserving the benefits of specialization.
Experiments on GAIA-text, WebWalkerQA, and FRAMES show that MATPO consistently
outperforms single-agent baselines by an average of 18.38% relative improvement
in performance and exhibits greater robustness to noisy tool outputs. Our
findings highlight the effectiveness of unifying multiple agent roles within a
single LLM and provide practical insights for stable and efficient multi-agent
RL training.
68e62542975ac4c405ef21eb
Zhanfeng Mo
claimed_verified
2025-10-16T10:38:49.298Z
63f0e78e6309c84d5f4c319b
/avatars/5af92388f562968b98dcab3e8ff729a6.svg
Zhanfeng Mo
user
mzf666
68e62542975ac4c405ef21ec
Xingxuan Li
68e62542975ac4c405ef21ed
Yuntao Chen
68e62542975ac4c405ef21ee
Lidong Bing
Multi-Agent Tool-Integrated Policy Optimization
Large language models (LLMs) increasingly rely on multi-turn tool-integrated
planning for knowledge-intensive and complex reasoning tasks. Existing
implementations typically rely on a single agent, but they suffer from limited
context length and noisy tool responses. A natural solution is to adopt a
multi-agent framework with planner- and worker-agents to manage context.
However, no existing methods support effective reinforcement learning
post-training of tool-integrated multi-agent frameworks. To address this gap,
we propose Multi-Agent Tool-Integrated Policy Optimization (MATPO), which
enables distinct roles (planner and worker) to be trained within a single LLM
instance using role-specific prompts via reinforcement learning. MATPO is
derived from a principled credit assignment mechanism across planner and worker
rollouts. This design eliminates the need to deploy multiple LLMs, which would
be memory-intensive, while preserving the benefits of specialization.
Experiments on GAIA-text, WebWalkerQA, and FRAMES show that MATPO consistently
outperforms single-agent baselines by an average of 18.38% relative improvement
in performance and exhibits greater robustness to noisy tool outputs. Our
findings highlight the effectiveness of unifying multiple agent roles within a
single LLM and provide practical insights for stable and efficient multi-agent
RL training.
MATPO, a reinforcement learning method, optimizes tool-integrated multi-agent roles within a single LLM, improving performance and robustness over single-agent systems.
multi-turn tool-integrated planning multi-agent framework planner-agents worker-agents reinforcement learning role-specific prompts credit assignment mechanism planner and worker rollouts multi-agent RL training