UserRL: Training Interactive User-Centric Agent via Reinforcement
  Learning
Reinforcement learning (RL) has shown promise in training agentic models that
move beyond static benchmarks to engage in dynamic, multi-turn interactions.
Yet, the ultimate value of such agents lies in their ability to assist users, a
setting where diversity and dynamics of user interaction pose challenges. In
this work, we propose UserRL, a unified framework for training and evaluating
user-centric abilities through standardized gym environments paired with
simulated users. We systematically vary turn-level reward assignment and
trajectory-level score calculation to analyze how different formulations affect
learning under the GRPO algorithm. Our experiments across Qwen3 models reveal
three key findings: (i) SFT cold start is critical for unlocking initial
interaction ability and enabling sustained RL improvements; (ii) deliberate
trajectory scoring yields more efficient and effective multi-turn interactions;
and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training,
open-source simulators (e.g., Qwen3-32B) remain a cost-effective and
transferable option. Together, these results highlight that careful design of
reward shaping and user simulation choice is as crucial as model scale, and
establish UserRL as a practical pathway for developing robust user-centric
agentic models. All codes and data are public for future research.
68d722840177a6054b013718
Cheng Qian
68d722840177a6054b013719
Zuxin Liu
68d722840177a6054b01371a
Akshara Prabhakar
68d722840177a6054b01371b
Jielin Qiu
68d722840177a6054b01371c
Zhiwei Liu
68d722840177a6054b01371d
Haolin Chen
68d722840177a6054b01371e
Shirley Kokane
68d722840177a6054b01371f
Heng Ji
68d722840177a6054b013720
Weiran Yao
68d722840177a6054b013721
Shelby Heinecke
68d722840177a6054b013722
Silvio Savarese
68d722840177a6054b013723
Caiming Xiong
68d722840177a6054b013724
Huan Wang
UserRL: Training Interactive User-Centric Agent via Reinforcement
  Learning
Reinforcement learning (RL) has shown promise in training agentic models that
move beyond static benchmarks to engage in dynamic, multi-turn interactions.
Yet, the ultimate value of such agents lies in their ability to assist users, a
setting where diversity and dynamics of user interaction pose challenges. In
this work, we propose UserRL, a unified framework for training and evaluating
user-centric abilities through standardized gym environments paired with
simulated users. We systematically vary turn-level reward assignment and
trajectory-level score calculation to analyze how different formulations affect
learning under the GRPO algorithm. Our experiments across Qwen3 models reveal
three key findings: (i) SFT cold start is critical for unlocking initial
interaction ability and enabling sustained RL improvements; (ii) deliberate
trajectory scoring yields more efficient and effective multi-turn interactions;
and (iii) while stronger simulated users (e.g., GPT-4o) facilitates training,
open-source simulators (e.g., Qwen3-32B) remain a cost-effective and
transferable option. Together, these results highlight that careful design of
reward shaping and user simulation choice is as crucial as model scale, and
establish UserRL as a practical pathway for developing robust user-centric
agentic models. All codes and data are public for future research.
UserRL framework enhances user-centric RL agents by optimizing reward assignment and user simulation, demonstrating the importance of these factors over model scale.
UserRL reinforcement learning agentic models dynamic interactions gym environments simulated users turn-level reward assignment trajectory-level score calculation GRPO algorithm Qwen3 models SFT cold start deliberate trajectory scoring multi-turn interactions open-source simulators