A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
We study what actually works and what doesn't for training large language
models as agents via multi-turn reinforcement learning. Despite rapid progress,
existing frameworks and definitions are fragmented, and there is no systematic
formulation or analysis of which design choices matter across tasks. We address
this gap by first breaking down the design space into three inter-related
pillars -- environment, reward, and policy -- and empirically derive a recipe
for training LLM agents in situated textual domains. In particular, we test
TextWorld and ALFWorld, popular domains for testing situated embodied
reasoning, as well as SWE-Gym for more software engineering style tasks. (i)
For the environment, we analyze the impacts of task complexity in terms of
sizes of the state and action spaces as well as optimal solution length,
finding that even simple environments within a domain can provide signal on how
well an agent can generalize to more complex tasks. (ii) For the reward, we
ablate relative reward sparsity, observing that while dense turn-level rewards
accelerate training, performance and stability is highly dependent on the
choice of RL algorithm. (iii) And for the agent's policy, we explore the
interplay between reward sparsity and biased (PPO, GRPO) and unbiased (RLOO)
policy gradient methods in addition to showing how to find the optimal
Supervised Fine-tuning (SFT) to RL training ratio given a fixed budget. We
distill these findings into a training recipe that guides co-design across the
three pillars, facilitating research and practical efforts in multi-turn
agentic RL. Code: https://github.com/pearls-lab/meow-tea-taro
68e36caf73e20ab5778420dc
Ruiyi Wang
68e36caf73e20ab5778420dd
Prithviraj Ammanabrolu
A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning
We study what actually works and what doesn't for training large language
models as agents via multi-turn reinforcement learning. Despite rapid progress,
existing frameworks and definitions are fragmented, and there is no systematic
formulation or analysis of which design choices matter across tasks. We address
this gap by first breaking down the design space into three inter-related
pillars -- environment, reward, and policy -- and empirically derive a recipe
for training LLM agents in situated textual domains. In particular, we test
TextWorld and ALFWorld, popular domains for testing situated embodied
reasoning, as well as SWE-Gym for more software engineering style tasks. (i)
For the environment, we analyze the impacts of task complexity in terms of
sizes of the state and action spaces as well as optimal solution length,
finding that even simple environments within a domain can provide signal on how
well an agent can generalize to more complex tasks. (ii) For the reward, we
ablate relative reward sparsity, observing that while dense turn-level rewards
accelerate training, performance and stability is highly dependent on the
choice of RL algorithm. (iii) And for the agent's policy, we explore the
interplay between reward sparsity and biased (PPO, GRPO) and unbiased (RLOO)
policy gradient methods in addition to showing how to find the optimal
Supervised Fine-tuning (SFT) to RL training ratio given a fixed budget. We
distill these findings into a training recipe that guides co-design across the
three pillars, facilitating research and practical efforts in multi-turn
agentic RL. Code: https://github.com/pearls-lab/meow-tea-taro
The study investigates effective design choices for training large language models as agents using multi-turn reinforcement learning across different environments and tasks, providing a comprehensive recipe for successful implementation.
multi-turn reinforcement learning large language models TextWorld ALFWorld SWE-Gym task complexity state space action space optimal solution length reward sparsity dense turn-level rewards RL algorithms policy gradient methods PPO GRPO RLOO Supervised Fine-tuning SFT RL training ratio