RL4F: Generating Natural Language Feedback with Reinforcement Learning
  for Repairing Model Outputs
Despite their unprecedented success, even the largest language models make
mistakes. Similar to how humans learn and improve using feedback, previous work
proposed providing language models with natural language feedback to guide them
in repairing their outputs. Because human-generated critiques are expensive to
obtain, researchers have devised learned critique generators in lieu of human
critics while assuming one can train downstream models to utilize generated
feedback. However, this approach does not apply to black-box or limited access
models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of
large general-purpose language agents, fine-tuning is neither computationally
nor spatially efficient as it results in multiple copies of the network. In
this work, we introduce RL4F (Reinforcement Learning for Feedback), a
multi-agent collaborative framework where the critique generator is trained to
maximize end-task performance of GPT-3, a fixed model more than 200 times its
size. RL4F produces critiques that help GPT-3 revise its outputs. We study
three datasets for action planning, summarization and alphabetization and show
improvements (~5% on average) in multiple text similarity metrics over strong
baselines across all three tasks.
6462e0b18bd7025ebe85297a
Afra Feyza Akyürek
extracted_confirmed
2024-01-23T18:14:37.075Z
617b009a7b4dce0224d6b8fb
/avatars/d42ebf8c75ec6236d9fcbb3d1e0ba0b7.svg
Afra Feyza Akyurek
user
feyzaakyurek
6462e0b18bd7025ebe85297b
Ekin Akyürek
6462e0b18bd7025ebe85297c
Aman Madaan
admin_assigned
2023-05-26T09:37:35.555Z
61bbc2b2d96f5d4111901dcb
/avatars/83c826d5cad7bab27d33585cb7e64575.svg
Aman Madaan
user
amanmadaan
6462e0b18bd7025ebe85297d
Ashwin Kalyan
6462e0b18bd7025ebe85297e
Peter Clark
6462e0b18bd7025ebe85297f
Derry Wijaya
6462e0b18bd7025ebe852980
Niket Tandon
admin_assigned
2023-05-26T09:38:29.616Z
62e69a709bb6d949274ad32b
/avatars/b6a796d12a1c1c708f5b46c9082cdee6.svg
Niket Tandon
user
nikett
RL4F: Generating Natural Language Feedback with Reinforcement Learning
  for Repairing Model Outputs
Despite their unprecedented success, even the largest language models make
mistakes. Similar to how humans learn and improve using feedback, previous work
proposed providing language models with natural language feedback to guide them
in repairing their outputs. Because human-generated critiques are expensive to
obtain, researchers have devised learned critique generators in lieu of human
critics while assuming one can train downstream models to utilize generated
feedback. However, this approach does not apply to black-box or limited access
models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of
large general-purpose language agents, fine-tuning is neither computationally
nor spatially efficient as it results in multiple copies of the network. In
this work, we introduce RL4F (Reinforcement Learning for Feedback), a
multi-agent collaborative framework where the critique generator is trained to
maximize end-task performance of GPT-3, a fixed model more than 200 times its
size. RL4F produces critiques that help GPT-3 revise its outputs. We study
three datasets for action planning, summarization and alphabetization and show
improvements (~5% on average) in multiple text similarity metrics over strong
baselines across all three tasks.
RL4F, a multi-agent framework, enhances GPT-3's performance through learned critiques generated to improve end-task performance.
Reinforcement Learning multi-agent collaborative framework critique generator GPT-3 action planning summarization alphabetization text similarity metrics