Stronger Together: On-Policy Reinforcement Learning for Collaborative
  LLMs
Multi-agent systems (MAS) and reinforcement learning (RL) are widely used to
enhance the agentic capabilities of large language models (LLMs). MAS improves
task performance through role-based orchestration, while RL uses environmental
rewards to learn stronger policies, such as GRPO-style optimization. However,
applying on-policy RL to MAS remains underexplored and presents unique
challenges. Algorithmically, standard GRPO grouping assumptions break down
because prompts vary by role and by turn. System-wise, the training stack must
support MAS-workflow rollouts and on-policy updates for both single-policy and
multi-policy models.
  We propose AT-GRPO, which includes (i) an agent- and turn-wise grouped RL
algorithm tailored to MAS and (ii) a training system that supports both single-
and multi-policy regimes. Across game, planning, coding, and math tasks,
AT-GRPO delivers substantial gains. On long-horizon planning, it increases
accuracy from a 14.0 to 47.0 percent single-agent RL baseline to 96.0 to 99.5
percent. It also improves reasoning performance, with average gains of 3.87 to
7.62 percent on coding tasks and 9.0 to 17.93 percent on math. Code and
environments are available at: https://github.com/pettingllms-ai/PettingLLMs.
68f089fb36f8b025381e1a3b
Yujie Zhao
claimed_verified
2025-10-17T04:13:58.637Z
65d4a6b48fa4ad07bc29e401
/avatars/5977aa13d73d07e25f436101b4bb363c.svg
Yujie Zhao
user
zhaozhaozhao04
68f089fb36f8b025381e1a3c
Lanxiang Hu
claimed_verified
2025-10-16T06:35:06.135Z
6301d6455e305a35cb0846a7
https://cdn-avatars.huggingface.co/v1/production/uploads/6301d6455e305a35cb0846a7/aT2AtzRMSY_T3y02MIUap.jpeg
Lanxiang Hu
user
Snyhlxde
68f089fb36f8b025381e1a3d
Yang Wang
68f089fb36f8b025381e1a3e
Minmin Hou
68f089fb36f8b025381e1a3f
Hao Zhang
68f089fb36f8b025381e1a40
Ke Ding
68f089fb36f8b025381e1a41
Jishen Zhao
Stronger Together: On-Policy Reinforcement Learning for Collaborative
  LLMs
Multi-agent systems (MAS) and reinforcement learning (RL) are widely used to
enhance the agentic capabilities of large language models (LLMs). MAS improves
task performance through role-based orchestration, while RL uses environmental
rewards to learn stronger policies, such as GRPO-style optimization. However,
applying on-policy RL to MAS remains underexplored and presents unique
challenges. Algorithmically, standard GRPO grouping assumptions break down
because prompts vary by role and by turn. System-wise, the training stack must
support MAS-workflow rollouts and on-policy updates for both single-policy and
multi-policy models.
  We propose AT-GRPO, which includes (i) an agent- and turn-wise grouped RL
algorithm tailored to MAS and (ii) a training system that supports both single-
and multi-policy regimes. Across game, planning, coding, and math tasks,
AT-GRPO delivers substantial gains. On long-horizon planning, it increases
accuracy from a 14.0 to 47.0 percent single-agent RL baseline to 96.0 to 99.5
percent. It also improves reasoning performance, with average gains of 3.87 to
7.62 percent on coding tasks and 9.0 to 17.93 percent on math. Code and
environments are available at: https://github.com/pettingllms-ai/PettingLLMs.
AT-GRPO, a tailored RL algorithm for multi-agent systems, significantly enhances performance across various tasks by addressing unique challenges in on-policy RL.
multi-agent systems reinforcement learning large language models role-based orchestration GRPO-style optimization on-policy RL agent-wise grouped RL turn-wise grouped RL single-policy multi-policy long-horizon planning reasoning performance coding tasks math tasks