Iterative Length-Regularized Direct Preference Optimization: A Case
  Study on Improving 7B Language Models to GPT-4 Level
Direct Preference Optimization (DPO), a standard method for aligning language
models with human preferences, is traditionally applied to offline preferences.
Recent studies show that DPO benefits from iterative training with online
preferences labeled by a trained reward model. In this work, we identify a
pitfall of vanilla iterative DPO - improved response quality can lead to
increased verbosity. To address this, we introduce iterative length-regularized
DPO (iLR-DPO) to penalize response length. Our empirical results show that
iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win
rate against GPT-4 Preview on AlpacaEval 2.0, and excels across
standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
These results demonstrate the effectiveness of iterative DPO in aligning
language models with human feedback.
6670f21456709bb492312de9
Jie Liu
claimed_verified
2024-06-18T09:11:05.671Z
639be86b59473c6ae02ef9c4
https://cdn-avatars.huggingface.co/v1/production/uploads/639be86b59473c6ae02ef9c4/gw34RBCVZCOkcAA79xUr3.png
Jie Liu
user
jieliu
6670f21456709bb492312dea
Zhanhui Zhou
admin_assigned
2024-06-21T12:56:23.932Z
642e5a7ba0b65dce1f87a7a2
/avatars/3ae01c9330a47e98fac9f1eb0ba94073.svg
Zhanhui Zhou
user
ZHZisZZ
6670f21456709bb492312deb
Jiaheng Liu
admin_assigned
2024-06-21T12:56:29.594Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
6670f21456709bb492312dec
Xingyuan Bu
admin_assigned
2024-06-21T12:56:35.212Z
6444e7765691ca69b0d95856
https://cdn-avatars.huggingface.co/v1/production/uploads/6444e7765691ca69b0d95856/WbtUXVbC2VIribtVAsLp7.jpeg
Xingyuan Bu
user
sefira32
6670f21456709bb492312ded
Chao Yang
6670f21456709bb492312dee
Han-Sen Zhong
admin_assigned
2024-06-21T12:56:42.468Z
663cc7a6662ad3bf0b346d78
/avatars/15e4e187225d25051d348f3f722041ca.svg
Han-Sen Zhong
user
hansenzhong
6670f21456709bb492312def
Wanli Ouyang
Iterative Length-Regularized Direct Preference Optimization: A Case
  Study on Improving 7B Language Models to GPT-4 Level
Direct Preference Optimization (DPO), a standard method for aligning language
models with human preferences, is traditionally applied to offline preferences.
Recent studies show that DPO benefits from iterative training with online
preferences labeled by a trained reward model. In this work, we identify a
pitfall of vanilla iterative DPO - improved response quality can lead to
increased verbosity. To address this, we introduce iterative length-regularized
DPO (iLR-DPO) to penalize response length. Our empirical results show that
iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win
rate against GPT-4 Preview on AlpacaEval 2.0, and excels across
standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
These results demonstrate the effectiveness of iterative DPO in aligning
language models with human feedback.
Iterative length-regularized DPO effectively aligns language models with human preferences while controlling verbosity, outperforming larger models on various benchmarks.
Direct Preference Optimization DPO iterative DPO iLR-DPO length-regularized response quality verbosity GPT-4 AlpacaEval 2.0 MT-Bench Arena-Hard OpenLLM Leaderboard