Iterative Length-Regularized Direct Preference Optimization: A Case
  Study on Improving 7B Language Models to GPT-4 Level
Direct Preference Optimization (DPO), a standard method for aligning language
models with human preferences, is traditionally applied to offline preferences.
Recent studies show that DPO benefits from iterative training with online
preferences labeled by a trained reward model. In this work, we identify a
pitfall of vanilla iterative DPO - improved response quality can lead to
increased verbosity. To address this, we introduce iterative length-regularized
DPO (iLR-DPO) to penalize response length. Our empirical results show that
iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win
rate against GPT-4 Preview on AlpacaEval 2.0, and excels across
standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
These results demonstrate the effectiveness of iterative DPO in aligning
language models with human feedback.
6670f21456709bb492312de9
Jie Liu
claimed_verified
2024-06-18T09:11:05.671Z
639be86b59473c6ae02ef9c4
https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
Jie Liu
user
jieliu
6670f21456709bb492312dea
Zhanhui Zhou
admin_assigned
2024-06-21T12:56:23.932Z
642e5a7ba0b65dce1f87a7a2
/avatars/3ae01c9330a47e98fac9f1eb0ba94073.svg
Zhanhui Zhou
user
ZHZisZZ
6670f21456709bb492312deb
Jiaheng Liu
admin_assigned
2024-06-21T12:56:29.594Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
6670f21456709bb492312dec
Xingyuan Bu
admin_assigned
2024-06-21T12:56:35.212Z
6444e7765691ca69b0d95856
/avatars/4ae5001e7c7dea89c4deaf2b05436857.svg
Xingyuan Bu
user
sefira32
6670f21456709bb492312ded
Chao Yang
6670f21456709bb492312dee
Han-Sen Zhong
admin_assigned
2024-06-21T12:56:42.468Z
663cc7a6662ad3bf0b346d78
/avatars/15e4e187225d25051d348f3f722041ca.svg
Han-Sen Zhong
user
hansenzhong
6670f21456709bb492312def
Wanli Ouyang
Iterative Length-Regularized Direct Preference Optimization: A Case
  Study on Improving 7B Language Models to GPT-4 Level
Direct Preference Optimization (DPO), a standard method for aligning language
models with human preferences, is traditionally applied to offline preferences.
Recent studies show that DPO benefits from iterative training with online
preferences labeled by a trained reward model. In this work, we identify a
pitfall of vanilla iterative DPO - improved response quality can lead to
increased verbosity. To address this, we introduce iterative length-regularized
DPO (iLR-DPO) to penalize response length. Our empirical results show that
iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win
rate against GPT-4 Preview on AlpacaEval 2.0, and excels across
standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
These results demonstrate the effectiveness of iterative DPO in aligning
language models with human feedback.
Direct Preference Optimization (DPO) offline preferences online preferences reward model iterative DPO length-regularized DPO (iLR-DPO) response length perplexity length-controlled win rate AlpacaEval 2.0 MT-Bench Arena-Hard OpenLLM Leaderboard human feedback