Fine-Grained Human Feedback Gives Better Rewards for Language Model
  Training
Language models (LMs) often exhibit undesirable text generation behaviors,
including generating false, toxic, or irrelevant outputs. Reinforcement
learning from human feedback (RLHF) - where human preference judgments on LM
outputs are transformed into a learning signal - has recently shown promise in
addressing these issues. However, such holistic feedback conveys limited
information on long text outputs; it does not indicate which aspects of the
outputs influenced user preference; e.g., which parts contain what type(s) of
errors. In this paper, we use fine-grained human feedback (e.g., which sentence
is false, which sub-sentence is irrelevant) as an explicit training signal. We
introduce Fine-Grained RLHF, a framework that enables training and learning
from reward functions that are fine-grained in two respects: (1) density,
providing a reward after every segment (e.g., a sentence) is generated; and (2)
incorporating multiple reward models associated with different feedback types
(e.g., factual incorrectness, irrelevance, and information incompleteness). We
conduct experiments on detoxification and long-form question answering to
illustrate how learning with such reward functions leads to improved
performance, supported by both automatic and human evaluation. Additionally, we
show that LM behaviors can be customized using different combinations of
fine-grained reward models. We release all data, collected human feedback, and
codes at https://FineGrainedRLHF.github.io.
647d58da36e109abce40dc99
Zeqiu Wu
extracted_pending
2023-06-05T03:39:07.589Z
647d335836e109abce3ffb79
/avatars/fee73f1e15502ca51f8a7a928f653d31.svg
Zeqiu Wu
user
zeqiuwu1
647d58da36e109abce40dc9a
Yushi Hu
extracted_confirmed
2023-06-05T09:56:38.574Z
62b1474bdcbad6848a91a54e
/avatars/d7308899b46232cad4a48a0e876449a8.svg
Yushi Hu
user
yushihu
647d58da36e109abce40dc9b
Weijia Shi
647d58da36e109abce40dc9c
Nouha Dziri
647d58da36e109abce40dc9d
Alane Suhr
claimed_verified
2024-06-09T20:08:55.528Z
6611e6e1188ff298b0dd0b79
/avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
Alane Suhr
user
alsuhr
647d58da36e109abce40dc9e
Prithviraj Ammanabrolu
647d58da36e109abce40dc9f
Noah A. Smith
647d58da36e109abce40dca0
Mari Ostendorf
647d58da36e109abce40dca1
Hannaneh Hajishirzi
Fine-Grained Human Feedback Gives Better Rewards for Language Model
  Training
Language models (LMs) often exhibit undesirable text generation behaviors,
including generating false, toxic, or irrelevant outputs. Reinforcement
learning from human feedback (RLHF) - where human preference judgments on LM
outputs are transformed into a learning signal - has recently shown promise in
addressing these issues. However, such holistic feedback conveys limited
information on long text outputs; it does not indicate which aspects of the
outputs influenced user preference; e.g., which parts contain what type(s) of
errors. In this paper, we use fine-grained human feedback (e.g., which sentence
is false, which sub-sentence is irrelevant) as an explicit training signal. We
introduce Fine-Grained RLHF, a framework that enables training and learning
from reward functions that are fine-grained in two respects: (1) density,
providing a reward after every segment (e.g., a sentence) is generated; and (2)
incorporating multiple reward models associated with different feedback types
(e.g., factual incorrectness, irrelevance, and information incompleteness). We
conduct experiments on detoxification and long-form question answering to
illustrate how learning with such reward functions leads to improved
performance, supported by both automatic and human evaluation. Additionally, we
show that LM behaviors can be customized using different combinations of
fine-grained reward models. We release all data, collected human feedback, and
codes at https://FineGrainedRLHF.github.io.
Fine-Grained Reinforcement Learning from Human Feedback improves language model performance by using detailed human feedback to provide segment-specific rewards for diverse feedback types.
Reinforcement learning from human feedback (RLHF) fine-grained human feedback Fine-Grained RLHF density rewards multiple reward models factual incorrectness irrelevance information incompleteness