SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning
  Tasks
Large language model (LLM) agents need to perform multi-turn interactions in
real-world tasks. However, existing multi-turn RL algorithms for optimizing LLM
agents fail to perform effective credit assignment over multiple turns while
leveraging the generalization capabilities of LLMs and it remains unclear how
to develop such algorithms. To study this, we first introduce a new benchmark,
ColBench, where an LLM agent interacts with a human collaborator over multiple
turns to solve realistic tasks in backend programming and frontend design.
Building on this benchmark, we propose a novel RL algorithm, SWEET-RL (RL with
Step-WisE Evaluation from Training-time information), that uses a carefully
designed optimization objective to train a critic model with access to
additional training-time information. The critic provides step-level rewards
for improving the policy model. Our experiments demonstrate that SWEET-RL
achieves a 6% absolute improvement in success and win rates on ColBench
compared to other state-of-the-art multi-turn RL algorithms, enabling
Llama-3.1-8B to match or exceed the performance of GPT4-o in realistic
collaborative content creation.
67db96db0d13633dcbe53fd6
Yifei Zhou
admin_assigned
2025-03-20T19:29:41.066Z
64fd0229e0dc35986bd3c0e5
/avatars/94f5698f9104dad7288edb4460026fd8.svg
Yifei Zhou
user
yifeizhou
67db96db0d13633dcbe53fd7
Song Jiang
admin_assigned
2025-03-20T19:30:15.739Z
629ff655198a9a5a7562d437
/avatars/42fd229a1521dc218c1fb385dd867b3b.svg
Song Jiang
user
songjiang
67db96db0d13633dcbe53fd8
Yuandong Tian
admin_assigned
2025-03-20T19:30:08.515Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
67db96db0d13633dcbe53fd9
Jason Weston
admin_assigned
2025-03-20T19:30:02.617Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
67db96db0d13633dcbe53fda
Sergey Levine
admin_assigned
2025-03-20T19:29:54.687Z
665ce54120a307a3754849dd
/avatars/e698726e9be61dd50ce2efe372ed5dac.svg
Sergey Levine
user
svlevine
67db96db0d13633dcbe53fdb
Sainbayar Sukhbaatar
admin_assigned
2025-03-20T19:29:48.384Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
67db96db0d13633dcbe53fdc
Xian Li
claimed_verified
2025-05-26T08:41:42.399Z
659a395421a7431643caedda
/avatars/c1e0bbcedce68fe3b4fe39e0cf01c65c.svg
Xian Li
user
xlxxl
SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning
  Tasks
Large language model (LLM) agents need to perform multi-turn interactions in
real-world tasks. However, existing multi-turn RL algorithms for optimizing LLM
agents fail to perform effective credit assignment over multiple turns while
leveraging the generalization capabilities of LLMs and it remains unclear how
to develop such algorithms. To study this, we first introduce a new benchmark,
ColBench, where an LLM agent interacts with a human collaborator over multiple
turns to solve realistic tasks in backend programming and frontend design.
Building on this benchmark, we propose a novel RL algorithm, SWEET-RL (RL with
Step-WisE Evaluation from Training-time information), that uses a carefully
designed optimization objective to train a critic model with access to
additional training-time information. The critic provides step-level rewards
for improving the policy model. Our experiments demonstrate that SWEET-RL
achieves a 6% absolute improvement in success and win rates on ColBench
compared to other state-of-the-art multi-turn RL algorithms, enabling
Llama-3.1-8B to match or exceed the performance of GPT4-o in realistic
collaborative content creation.
A novel RL algorithm, SWEET-RL, uses step-level rewards to enhance multi-turn interactions in LLMs, outperforming existing methods on a new benchmark for collaborative content creation tasks.
LLM multi-turn RL credit assignment ColBench backend programming frontend design critic model training-time information step-level rewards policy model success rates win rates GPT4-o