Contrastive Prefence Learning: Learning from Human Feedback without RL
Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular
paradigm for aligning models with human intent. Typically RLHF algorithms
operate in two phases: first, use human preferences to learn a reward function
and second, align the model by optimizing the learned reward via reinforcement
learning (RL). This paradigm assumes that human preferences are distributed
according to reward, but recent work suggests that they instead follow the
regret under the user's optimal policy. Thus, learning a reward function from
feedback is not only based on a flawed assumption of human preference, but also
leads to unwieldy optimization challenges that stem from policy gradients or
bootstrapping in the RL phase. Because of these optimization challenges,
contemporary RLHF methods restrict themselves to contextual bandit settings
(e.g., as in large language models) or limit observation dimensionality (e.g.,
state-based robotics). We overcome these limitations by introducing a new
family of algorithms for optimizing behavior from human feedback using the
regret-based model of human preferences. Using the principle of maximum
entropy, we derive Contrastive Preference Learning (CPL), an algorithm for
learning optimal policies from preferences without learning reward functions,
circumventing the need for RL. CPL is fully off-policy, uses only a simple
contrastive objective, and can be applied to arbitrary MDPs. This enables CPL
to elegantly scale to high-dimensional and sequential RLHF problems while being
simpler than prior methods.
6535f1a125a58aec65c6ac6f
Joey Hejna
6535f1a125a58aec65c6ac70
Rafael Rafailov
admin_assigned
2023-10-23T08:43:41.615Z
64701c98d742e9ef6522598f
/avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
Rafael Rafailov
user
rmrafailov
6535f1a125a58aec65c6ac71
Harshit Sikchi
extracted_confirmed
2023-10-23T04:22:55.676Z
6304559b1dd5d3c6248878f1
/avatars/50ad7ffcc226c9ed7c6a78634a5fc8b4.svg
Harshit Sikchi
user
hsikchi
6535f1a125a58aec65c6ac72
Chelsea Finn
admin_assigned
2023-10-23T08:43:48.764Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
6535f1a125a58aec65c6ac73
Scott Niekum
claimed_verified
2023-10-23T14:46:38.498Z
653675a67f86289757f461ca
/avatars/47fe2c67305a3062c29c27aa2ae642b2.svg
Scott Niekum
user
sniekum
6535f1a125a58aec65c6ac74
W. Bradley Knox
claimed_verified
2023-10-24T07:29:47.953Z
64aee90a710608ee1a01b974
/avatars/7ea57ca3a4c75ad714a8b5d3f712c4c6.svg
Brad Knox
user
wbknox
6535f1a125a58aec65c6ac75
Dorsa Sadigh
Contrastive Prefence Learning: Learning from Human Feedback without RL
Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular
paradigm for aligning models with human intent. Typically RLHF algorithms
operate in two phases: first, use human preferences to learn a reward function
and second, align the model by optimizing the learned reward via reinforcement
learning (RL). This paradigm assumes that human preferences are distributed
according to reward, but recent work suggests that they instead follow the
regret under the user's optimal policy. Thus, learning a reward function from
feedback is not only based on a flawed assumption of human preference, but also
leads to unwieldy optimization challenges that stem from policy gradients or
bootstrapping in the RL phase. Because of these optimization challenges,
contemporary RLHF methods restrict themselves to contextual bandit settings
(e.g., as in large language models) or limit observation dimensionality (e.g.,
state-based robotics). We overcome these limitations by introducing a new
family of algorithms for optimizing behavior from human feedback using the
regret-based model of human preferences. Using the principle of maximum
entropy, we derive Contrastive Preference Learning (CPL), an algorithm for
learning optimal policies from preferences without learning reward functions,
circumventing the need for RL. CPL is fully off-policy, uses only a simple
contrastive objective, and can be applied to arbitrary MDPs. This enables CPL
to elegantly scale to high-dimensional and sequential RLHF problems while being
simpler than prior methods.
A new regret-based algorithm, Contrastive Preference Learning (CPL), learns optimal policies directly from human preferences without learning a reward function, addressing optimization challenges in Reinforcement Learning from Human Feedback (RLHF).
Reinforcement Learning from Human Feedback RLHF reward function human preferences regret maximum entropy Contrastive Preference Learning CPL off-policy contrastive objective MDPs