Learn Your Reference Model for Real Good Alignment
The complexity of the alignment problem stems from the fact that existing
methods are unstable. Researchers continuously invent various tricks to address
this shortcoming. For instance, in the fundamental Reinforcement Learning From
Human Feedback (RLHF) technique of Language Model alignment, in addition to
reward maximization, the Kullback-Leibler divergence between the trainable
policy and the SFT policy is minimized. This addition prevents the model from
being overfitted to the Reward Model (RM) and generating texts that are
out-of-domain for the RM. The Direct Preference Optimization (DPO) method
reformulates the optimization task of RLHF and eliminates the Reward Model
while tacitly maintaining the requirement for the policy to be close to the SFT
policy. In our paper, we argue that this implicit limitation in the DPO method
leads to sub-optimal results. We propose a new method called Trust Region DPO
(TR-DPO), which updates the reference policy during training. With such a
straightforward update, we demonstrate the effectiveness of TR-DPO against DPO
on the Anthropic HH and TLDR datasets. We show that TR-DPO outperforms DPO by
up to 19%, measured by automatic evaluation with GPT-4. The new alignment
approach that we propose allows us to improve the quality of models across
several parameters at once, such as coherence, correctness, level of detail,
helpfulness, and harmlessness.
661de9b836eb62686fcce152
Alexey Gorbatovski
claimed_verified
2024-04-16T08:59:53.613Z
62897fce5d9e25c10e4f319d
https://cdn-avatars.huggingface.co/v1/production/uploads/62897fce5d9e25c10e4f319d/bMlfAyzkNNZlkQ5mCW6Vc.jpeg
Alexey Gorbatovski
user
Myashka
661de9b836eb62686fcce153
Boris Shaposhnikov
claimed_verified
2024-04-16T09:04:54.014Z
637dd11dcbad6e62a5e39743
https://cdn-avatars.huggingface.co/v1/production/uploads/637dd11dcbad6e62a5e39743/DG3rM8cy8inqbCoG4qizO.jpeg
Boris Shaposhnikov
user
borisshapa
661de9b836eb62686fcce154
Alexey Malakhov
claimed_verified
2024-04-16T10:43:31.078Z
636e71b2b0ebc04888157b71
/avatars/957ba705d470e3a01792741d7f0ff038.svg
Alexey Malakhov
user
ZeL1k7
661de9b836eb62686fcce155
Nikita Surnachev
661de9b836eb62686fcce156
Yaroslav Aksenov
claimed_verified
2024-04-16T23:16:54.902Z
63ed5676684767daecac6f8a
/avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
Yaroslav Aksenov
user
yaraksen
661de9b836eb62686fcce157
Ian Maksimov
claimed_verified
2024-04-16T12:44:33.851Z
65cbcb760b3bd8f5ce402bd5
/avatars/2a60e344cf219eed58217ccf5db26680.svg
Ian Maksimov
user
BayesianMonster
661de9b836eb62686fcce158
Nikita Balagansky
claimed_verified
2024-04-16T08:20:16.276Z
60b364e7f88532cd79eaff7b
https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
Nikita Balagansky
user
elephantmipt
661de9b836eb62686fcce159
Daniil Gavrilov
claimed_verified
2024-04-16T08:15:54.509Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
Learn Your Reference Model for Real Good Alignment
The complexity of the alignment problem stems from the fact that existing
methods are unstable. Researchers continuously invent various tricks to address
this shortcoming. For instance, in the fundamental Reinforcement Learning From
Human Feedback (RLHF) technique of Language Model alignment, in addition to
reward maximization, the Kullback-Leibler divergence between the trainable
policy and the SFT policy is minimized. This addition prevents the model from
being overfitted to the Reward Model (RM) and generating texts that are
out-of-domain for the RM. The Direct Preference Optimization (DPO) method
reformulates the optimization task of RLHF and eliminates the Reward Model
while tacitly maintaining the requirement for the policy to be close to the SFT
policy. In our paper, we argue that this implicit limitation in the DPO method
leads to sub-optimal results. We propose a new method called Trust Region DPO
(TR-DPO), which updates the reference policy during training. With such a
straightforward update, we demonstrate the effectiveness of TR-DPO against DPO
on the Anthropic HH and TLDR datasets. We show that TR-DPO outperforms DPO by
up to 19%, measured by automatic evaluation with GPT-4. The new alignment
approach that we propose allows us to improve the quality of models across
several parameters at once, such as coherence, correctness, level of detail,
helpfulness, and harmlessness.
A new method, Trust Region DPO (TR-DPO), is proposed to improve policy alignment in reinforcement learning, outperforming Direct Preference Optimization (DPO) by up to 19% on key datasets by updating the reference policy during training.
Reinforcement Learning From Human Feedback (RLHF) Kullback-Leibler divergence reward maximization SFT policy Reward Model (RM) Direct Preference Optimization (DPO) Trust Region DPO (TR-DPO) Anthropic HH TLDR datasets GPT-4 coherence correctness level of detail helpfulness harmlessness