Segmenting Text and Learning Their Rewards for Improved RLHF in Language
  Model
Reinforcement learning from human feedback (RLHF) has been widely adopted to
align language models (LMs) with human preference. Prior RLHF works typically
take a bandit formulation, which, though intuitive, ignores the sequential
nature of LM generation and can suffer from the sparse reward issue. While
recent works propose dense token-level RLHF, treating each token as an action
may be oversubtle to proper reward assignment. In this paper, we seek to get
the best of both by training and utilizing a segment-level reward model, which
assigns a reward to each semantically complete text segment that spans over a
short sequence of tokens. For reward learning, our method allows dynamic text
segmentation and compatibility with standard sequence-preference datasets. For
effective RL-based LM training against segment reward, we generalize the
classical scalar bandit reward normalizers into location-aware normalizer
functions and interpolate the segment reward for further densification. With
these designs, our method performs competitively on three popular RLHF
benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation
studies are conducted to further demonstrate our method.
677dea23e86d0754dc6e3f09
Yueqin Yin
claimed_verified
2025-01-08T09:44:41.339Z
605e8dfd5abeb13e714c4c18
/avatars/bc27a0ed17b2bd4311e89d3028fa327b.svg
yueqin yin
user
yyqoni
677dea23e86d0754dc6e3f0a
Shentao Yang
claimed_verified
2025-01-08T21:08:25.725Z
677ebaf3fcaae73ddd6c8475
/avatars/98db20fe3bbbf7caffb5821a7242dc54.svg
Shentao Yang
user
shentaoyang
677dea23e86d0754dc6e3f0b
Yujia Xie
677dea23e86d0754dc6e3f0c
Ziyi Yang
677dea23e86d0754dc6e3f0d
Yuting Sun
677dea23e86d0754dc6e3f0e
Hany Awadalla
677dea23e86d0754dc6e3f0f
Weizhu Chen
677dea23e86d0754dc6e3f10
Mingyuan Zhou
Segmenting Text and Learning Their Rewards for Improved RLHF in Language
  Model
Reinforcement learning from human feedback (RLHF) has been widely adopted to
align language models (LMs) with human preference. Prior RLHF works typically
take a bandit formulation, which, though intuitive, ignores the sequential
nature of LM generation and can suffer from the sparse reward issue. While
recent works propose dense token-level RLHF, treating each token as an action
may be oversubtle to proper reward assignment. In this paper, we seek to get
the best of both by training and utilizing a segment-level reward model, which
assigns a reward to each semantically complete text segment that spans over a
short sequence of tokens. For reward learning, our method allows dynamic text
segmentation and compatibility with standard sequence-preference datasets. For
effective RL-based LM training against segment reward, we generalize the
classical scalar bandit reward normalizers into location-aware normalizer
functions and interpolate the segment reward for further densification. With
these designs, our method performs competitively on three popular RLHF
benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation
studies are conducted to further demonstrate our method.
A segment-level reward model for reinforcement learning from human feedback addresses sparse reward issues by assigning rewards to semantically complete text segments during language model training.
reinforcement learning from human feedback bandit formulation sparse reward dense token-level RLHF segment-level reward model dynamic text segmentation sequence-preference datasets scalar bandit reward normalizers location-aware normalizer functions segment reward AlpacaEval 2.0 Arena-Hard MT-Bench ablation studies