RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn
  Reinforcement Learning
Training large language models (LLMs) as interactive agents presents unique
challenges including long-horizon decision making and interacting with
stochastic environment feedback. While reinforcement learning (RL) has enabled
progress in static tasks, multi-turn agent RL training remains underexplored.
We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a
general framework for trajectory-level agent RL, and introduce RAGEN, a modular
system for training and evaluating LLM agents. Our study on three stylized
environments reveals three core findings. First, our agent RL training shows a
recurring mode of Echo Trap where reward variance cliffs and gradient spikes;
we address this with StarPO-S, a stabilized variant with trajectory filtering,
critic incorporation, and decoupled clipping. Second, we find the shaping of RL
rollouts would benefit from diverse initial states, medium interaction
granularity and more frequent sampling. Third, we show that without
fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge
through multi-turn RL and they may show shallow strategies or hallucinated
thoughts. Code and environments are available at
https://github.com/RAGEN-AI/RAGEN.
68124aad34af9c89c46a800b
Zihan Wang
claimed_verified
2025-05-02T06:36:32.957Z
63318e014bd187e9ce05cec6
https://cdn-avatars.huggingface.co/v1/production/uploads/63318e014bd187e9ce05cec6/rKN84BAXF_u0osSu6KJy5.jpeg
Zihan Wang
user
ZihanWang314
68124aad34af9c89c46a800c
Kangrui Wang
68124aad34af9c89c46a800d
Qineng Wang
68124aad34af9c89c46a800e
Pingyue Zhang
68124aad34af9c89c46a800f
Linjie Li
68124aad34af9c89c46a8010
Zhengyuan Yang
68124aad34af9c89c46a8011
Kefan Yu
68124aad34af9c89c46a8012
Minh Nhat Nguyen
claimed_verified
2025-06-11T20:21:28.260Z
643957de21221ac7411789fd
/avatars/2857eb85988094f245e8e3eb1db34459.svg
Nguyen Nhat Minh
user
menhguin
68124aad34af9c89c46a8013
Licheng Liu
claimed_verified
2025-05-02T06:36:30.696Z
661caf2f6158c41091b3954e
https://cdn-avatars.huggingface.co/v1/production/uploads/661caf2f6158c41091b3954e/zrAqbiL4u1l6_Loea00_k.jpeg
Licheng Liu
user
LichengLiu03
68124aad34af9c89c46a8014
Eli Gottlieb
68124aad34af9c89c46a8015
Monica Lam
68124aad34af9c89c46a8016
Yiping Lu
68124aad34af9c89c46a8017
Kyunghyun Cho
68124aad34af9c89c46a8018
Jiajun Wu
68124aad34af9c89c46a8019
Li Fei-Fei
68124aad34af9c89c46a801a
Lijuan Wang
68124aad34af9c89c46a801b
Yejin Choi
68124aad34af9c89c46a801c
Manling Li
RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn
  Reinforcement Learning
Training large language models (LLMs) as interactive agents presents unique
challenges including long-horizon decision making and interacting with
stochastic environment feedback. While reinforcement learning (RL) has enabled
progress in static tasks, multi-turn agent RL training remains underexplored.
We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a
general framework for trajectory-level agent RL, and introduce RAGEN, a modular
system for training and evaluating LLM agents. Our study on three stylized
environments reveals three core findings. First, our agent RL training shows a
recurring mode of Echo Trap where reward variance cliffs and gradient spikes;
we address this with StarPO-S, a stabilized variant with trajectory filtering,
critic incorporation, and decoupled clipping. Second, we find the shaping of RL
rollouts would benefit from diverse initial states, medium interaction
granularity and more frequent sampling. Third, we show that without
fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge
through multi-turn RL and they may show shallow strategies or hallucinated
thoughts. Code and environments are available at
https://github.com/RAGEN-AI/RAGEN.
StarPO and RAGEN address challenges in training interactive language model agents through RL, improving stability and reward shaping in diverse environments.
StarPO RAGEN trajectory-level agent RL Echo Trap trajectory filtering critic incorporation decoupled clipping RL rollouts initial states interaction granularity sampling reasoning-aware reward signals agent reasoning shallow strategies hallucinated thoughts