The Era of Real-World Human Interaction: RL from User Conversations
We posit that to achieve continual model improvement and multifaceted
alignment, future models must learn from natural human interaction. Current
conversational models are aligned using pre-annotated, expert-generated human
feedback. In this work, we introduce Reinforcement Learning from Human
Interaction (RLHI), a paradigm that learns directly from in-the-wild user
conversations. We develop two complementary methods: (1) RLHI with User-Guided
Rewrites, which revises unsatisfactory model outputs based on users'
natural-language follow-up responses, (2) RLHI with User-Based Rewards, which
learns via a reward model conditioned on knowledge of the user's long-term
interaction history (termed persona). Together, these methods link long-term
user personas to turn-level preferences via persona-conditioned preference
optimization. Trained on conversations derived from WildChat, both RLHI
variants outperform strong baselines in personalization and
instruction-following, and similar feedback enhances performance on reasoning
benchmarks. These results suggest organic human interaction offers scalable,
effective supervision for personalized alignment.
68dbd78dd2bf1f4b15ec7af3
Chuanyang Jin
claimed_verified
2025-09-30T13:53:01.085Z
64beb69801f1983a86a05de2
https://cdn-avatars.huggingface.co/v1/production/uploads/64beb69801f1983a86a05de2/tFyCoqZ6gT8NWkZfuncID.jpeg
Chuanyang Jin
user
Chuanyang-Jin
68dbd78dd2bf1f4b15ec7af4
Jing Xu
68dbd78dd2bf1f4b15ec7af5
Bo Liu
claimed_verified
2025-10-01T10:24:29.731Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
user
Benjamin-eecs
68dbd78dd2bf1f4b15ec7af6
Leitian Tao
claimed_verified
2025-10-10T09:46:39.610Z
655eca47e1b6d15cfe8f8235
/avatars/1ab847d12957a8ab77c84bdc452e3f5f.svg
Tao
user
Leitian
68dbd78dd2bf1f4b15ec7af7
Olga Golovneva
68dbd78dd2bf1f4b15ec7af8
Tianmin Shu
68dbd78dd2bf1f4b15ec7af9
Wenting Zhao
68dbd78dd2bf1f4b15ec7afa
Xian Li
68dbd78dd2bf1f4b15ec7afb
Jason Weston
The Era of Real-World Human Interaction: RL from User Conversations
We posit that to achieve continual model improvement and multifaceted
alignment, future models must learn from natural human interaction. Current
conversational models are aligned using pre-annotated, expert-generated human
feedback. In this work, we introduce Reinforcement Learning from Human
Interaction (RLHI), a paradigm that learns directly from in-the-wild user
conversations. We develop two complementary methods: (1) RLHI with User-Guided
Rewrites, which revises unsatisfactory model outputs based on users'
natural-language follow-up responses, (2) RLHI with User-Based Rewards, which
learns via a reward model conditioned on knowledge of the user's long-term
interaction history (termed persona). Together, these methods link long-term
user personas to turn-level preferences via persona-conditioned preference
optimization. Trained on conversations derived from WildChat, both RLHI
variants outperform strong baselines in personalization and
instruction-following, and similar feedback enhances performance on reasoning
benchmarks. These results suggest organic human interaction offers scalable,
effective supervision for personalized alignment.
Reinforcement Learning from Human Interaction (RLHI) uses in-the-wild user conversations to improve conversational models, enhancing personalization and instruction-following through user-guided rewrites and persona-conditioned rewards.
Reinforcement Learning from Human Interaction (RLHI) User-Guided Rewrites User-Based Rewards persona-conditioned preference optimization WildChat