Language Models Can Learn from Verbal Feedback Without Scalar Rewards
LLMs are often trained with RL from human or AI feedback, yet such methods
typically compress nuanced feedback into scalar rewards, discarding much of
their richness and inducing scale imbalance. We propose treating verbal
feedback as a conditioning signal. Inspired by language priors in text-to-image
generation, which enable novel outputs from unseen prompts, we introduce the
feedback-conditional policy (FCP). FCP learns directly from response-feedback
pairs, approximating the feedback-conditional posterior through maximum
likelihood training on offline data. We further develop an online bootstrapping
stage where the policy generates under positive conditions and receives fresh
feedback to refine itself. This reframes feedback-driven learning as
conditional generation rather than reward optimization, offering a more
expressive way for LLMs to directly learn from verbal feedback. Our code is
available at https://github.com/sail-sg/feedback-conditional-policy.
68d9ef140177a6054b013acd
Renjie Luo
admin_assigned
2025-09-29T13:14:39.469Z
66f2aa5dc13823293e03b9bb
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7ggb86W70FZy1iPE4IVAY.png
RenJie
user
LuoRenjie
68d9ef140177a6054b013ace
Zichen Liu
claimed_verified
2025-10-02T03:09:16.197Z
65f5392c68b8e0cb3c9977a2
/avatars/aa64772475098e8a135c13072fde6744.svg
Zichen
user
lkevinzc
68d9ef140177a6054b013acf
Xiangyan Liu
68d9ef140177a6054b013ad0
Chao Du
68d9ef140177a6054b013ad1
Min Lin
68d9ef140177a6054b013ad2
Wenhu Chen
admin_assigned
2025-09-29T13:14:20.560Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
68d9ef140177a6054b013ad3
Wei Lu
68d9ef140177a6054b013ad4
Tianyu Pang
claimed_verified
2025-09-30T06:36:32.027Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
Language Models Can Learn from Verbal Feedback Without Scalar Rewards
LLMs are often trained with RL from human or AI feedback, yet such methods
typically compress nuanced feedback into scalar rewards, discarding much of
their richness and inducing scale imbalance. We propose treating verbal
feedback as a conditioning signal. Inspired by language priors in text-to-image
generation, which enable novel outputs from unseen prompts, we introduce the
feedback-conditional policy (FCP). FCP learns directly from response-feedback
pairs, approximating the feedback-conditional posterior through maximum
likelihood training on offline data. We further develop an online bootstrapping
stage where the policy generates under positive conditions and receives fresh
feedback to refine itself. This reframes feedback-driven learning as
conditional generation rather than reward optimization, offering a more
expressive way for LLMs to directly learn from verbal feedback. Our code is
available at https://github.com/sail-sg/feedback-conditional-policy.
Feedback-conditional policy (FCP) enables LLMs to learn from verbal feedback by treating it as a conditioning signal, improving expressiveness over scalar rewards.
LLMs RL human feedback AI feedback scalar rewards feedback-conditional policy language priors text-to-image generation response-feedback pairs maximum likelihood training offline data online bootstrapping conditional generation reward optimization