RLVF: Learning from Verbal Feedback without Overgeneralization
The diversity of contexts in which large language models (LLMs) are deployed
requires the ability to modify or customize default model behaviors to
incorporate nuanced requirements and preferences. A convenient interface to
specify such model adjustments is high-level verbal feedback, such as "Don't
use emojis when drafting emails to my boss." However, while writing high-level
feedback is far simpler than collecting annotations for reinforcement learning
from human feedback (RLHF), we find that simply prompting a model with such
feedback leads to overgeneralization of the feedback to contexts where it is
not relevant. We study the problem of incorporating verbal feedback without
such overgeneralization, inspiring a new method Contextualized Critiques with
Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level
feedback to generate a small synthetic preference dataset specifying how the
feedback should (and should not) be applied. It then fine-tunes the model in
accordance with the synthetic preference data while minimizing the divergence
from the original model for prompts where the feedback does not apply. Our
experimental results indicate that our approach effectively applies verbal
feedback to relevant scenarios while preserving existing behaviors for other
contexts. For both human- and GPT-4-generated high-level feedback, C3PO
effectively adheres to the given feedback comparably to in-context baselines
while reducing overgeneralization by 30%.
65d2bc1b351255ba48d156a7
Moritz Stephan
extracted_confirmed
2024-02-19T02:27:17.754Z
65263403ab48a81abd5847d5
/avatars/944765111baffb8a24a5b32fc3daab15.svg
Moritz Stephan
user
Moritz-Stephan
65d2bc1b351255ba48d156a8
Alexander Khazatsky
65d2bc1b351255ba48d156a9
Eric Mitchell
admin_assigned
2024-02-19T15:24:54.095Z
6514e9cbfc2301d62d7bf2ce
/avatars/eba6bd6e2840d7b5dd735e6662020d0c.svg
Eric mitchell
user
Pain777
65d2bc1b351255ba48d156aa
Annie S Chen
admin_assigned
2024-02-19T15:25:34.622Z
64c06c248137192a1e1e1428
/avatars/a1fc70a8f28a45807beec09e413ee362.svg
Annie Chen
user
anniesch
65d2bc1b351255ba48d156ab
Sheryl Hsu
65d2bc1b351255ba48d156ac
Archit Sharma
65d2bc1b351255ba48d156ad
Chelsea Finn
admin_assigned
2024-02-19T15:26:14.353Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
RLVF: Learning from Verbal Feedback without Overgeneralization
The diversity of contexts in which large language models (LLMs) are deployed
requires the ability to modify or customize default model behaviors to
incorporate nuanced requirements and preferences. A convenient interface to
specify such model adjustments is high-level verbal feedback, such as "Don't
use emojis when drafting emails to my boss." However, while writing high-level
feedback is far simpler than collecting annotations for reinforcement learning
from human feedback (RLHF), we find that simply prompting a model with such
feedback leads to overgeneralization of the feedback to contexts where it is
not relevant. We study the problem of incorporating verbal feedback without
such overgeneralization, inspiring a new method Contextualized Critiques with
Constrained Preference Optimization (C3PO). C3PO uses a piece of high-level
feedback to generate a small synthetic preference dataset specifying how the
feedback should (and should not) be applied. It then fine-tunes the model in
accordance with the synthetic preference data while minimizing the divergence
from the original model for prompts where the feedback does not apply. Our
experimental results indicate that our approach effectively applies verbal
feedback to relevant scenarios while preserving existing behaviors for other
contexts. For both human- and GPT-4-generated high-level feedback, C3PO
effectively adheres to the given feedback comparably to in-context baselines
while reducing overgeneralization by 30%.
A new method called C3PO is introduced to incorporate verbal feedback into large language models without overgeneralization by generating a synthetic preference dataset and fine-tuning the model accordingly.
large language models reinforcement learning from human feedback RLHF high-level verbal feedback overgeneralization Contextualized Critiques with Constrained Preference Optimization C3PO synthetic preference dataset fine-tuning divergence GPT-4