Dataset Reset Policy Optimization for RLHF
Reinforcement Learning (RL) from Human Preference-based feedback is a popular
paradigm for fine-tuning generative models, which has produced impressive
models such as GPT-4 and Claude3 Opus. This framework often consists of two
steps: learning a reward model from an offline preference dataset followed by
running online RL to optimize the learned reward model. In this work,
leveraging the idea of reset, we propose a new RLHF algorithm with provable
guarantees. Motivated by the fact that offline preference dataset provides
informative states (i.e., data that is preferred by the labelers), our new
algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing
offline preference dataset into the online policy training procedure via
dataset reset: it directly resets the policy optimizer to the states in the
offline dataset, instead of always starting from the initial state
distribution. In theory, we show that DR-PO learns to perform at least as good
as any policy that is covered by the offline dataset under general function
approximation with finite sample complexity. In experiments, we demonstrate
that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH)
dataset, the generation from DR-PO is better than that from Proximal Policy
Optimization (PPO) and Direction Preference Optimization (DPO), under the
metric of GPT4 win-rate. Code for this work can be found at
https://github.com/Cornell-RL/drpo.
661c8966c42687695eba6e2d
Jonathan D. Chang
661c8966c42687695eba6e2e
Wenhao Shan
extracted_confirmed
2024-05-01T03:45:42.291Z
6631baa4a8374cabcef53de1
/avatars/ab4f94cdd4104694d170646ed7463fa6.svg
Wenhao Zhan
user
whzhan
661c8966c42687695eba6e2f
Owen Oertell
claimed_verified
2024-04-16T00:02:57.340Z
65786ad2821f439498e5d584
/avatars/1a85ba5b7f6c6df32d12397057b04566.svg
Owen Oertell
user
ojo2
661c8966c42687695eba6e30
Kianté Brantley
claimed_verified
2024-04-19T07:39:06.629Z
644eb204722da4dfd871f1fa
/avatars/e09550b70161f8e541b91a019a4702b9.svg
kiante
user
xkianteb
661c8966c42687695eba6e31
Dipendra Misra
661c8966c42687695eba6e32
Jason D. Lee
extracted_pending
2024-04-15T01:56:55.265Z
64ad70f7f9f761a4c3910c63
/avatars/69b6c167299ca7a174a69ddc5a95e4a2.svg
Jason Lee
user
Jasondlee
661c8966c42687695eba6e33
Wen Sun
Dataset Reset Policy Optimization for RLHF
Reinforcement Learning (RL) from Human Preference-based feedback is a popular
paradigm for fine-tuning generative models, which has produced impressive
models such as GPT-4 and Claude3 Opus. This framework often consists of two
steps: learning a reward model from an offline preference dataset followed by
running online RL to optimize the learned reward model. In this work,
leveraging the idea of reset, we propose a new RLHF algorithm with provable
guarantees. Motivated by the fact that offline preference dataset provides
informative states (i.e., data that is preferred by the labelers), our new
algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing
offline preference dataset into the online policy training procedure via
dataset reset: it directly resets the policy optimizer to the states in the
offline dataset, instead of always starting from the initial state
distribution. In theory, we show that DR-PO learns to perform at least as good
as any policy that is covered by the offline dataset under general function
approximation with finite sample complexity. In experiments, we demonstrate
that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH)
dataset, the generation from DR-PO is better than that from Proximal Policy
Optimization (PPO) and Direction Preference Optimization (DPO), under the
metric of GPT4 win-rate. Code for this work can be found at
https://github.com/Cornell-RL/drpo.
The DR-PO algorithm leverages reset techniques in RLHF to improve policy optimization by incorporating an offline preference dataset, resulting in better generative performance compared to PPO and DPO.
Reinforcement Learning RLHF reward model offline preference dataset policy optimizer Dataset Reset Policy Optimization DR-PO TL;DR summarization Anthropic Helpful Harmful Proximal Policy Optimization Direction Preference Optimization GPT4 win-rate