Pre-DPO: Improving Data Utilization in Direct Preference Optimization
  Using a Guiding Reference Model
Direct Preference Optimization (DPO) simplifies reinforcement learning from
human feedback (RLHF) for large language models (LLMs) by directly optimizing
human preferences without an explicit reward model. We find that during DPO
training, the reference model plays the role of a data weight adjuster.
However, the common practice of initializing the policy and reference models
identically in DPO can lead to inefficient data utilization and impose a
performance ceiling. Meanwhile, the lack of a reference model in Simple
Preference Optimization (SimPO) reduces training robustness and necessitates
stricter conditions to prevent catastrophic forgetting. In this work, we
propose Pre-DPO, a simple yet effective DPO-based training paradigm that
enhances preference optimization performance by leveraging a guiding reference
model. This reference model provides foresight into the optimal policy state
achievable through the training preference data, serving as a guiding mechanism
that adaptively assigns higher weights to samples more suitable for the model
and lower weights to those less suitable. Extensive experiments on AlpacaEval
2.0 and Arena-Hard v0.1 benchmarks demonstrate that Pre-DPO consistently
improves the performance of both DPO and SimPO, without relying on external
models or additional data.
6809948944114def75aaeb7d
Junshu Pan
6809948944114def75aaeb7e
Wei Shen
admin_assigned
2025-04-24T11:26:03.812Z
6468823272d9180d4ac90bdf
/avatars/70cb7d65d30ecb944595000ceeeedb1b.svg
Wei Shen
user
Swtheking
6809948944114def75aaeb7f
Shulin Huang
6809948944114def75aaeb80
Qiji Zhou
6809948944114def75aaeb81
Yue Zhang
Pre-DPO: Improving Data Utilization in Direct Preference Optimization
  Using a Guiding Reference Model
Direct Preference Optimization (DPO) simplifies reinforcement learning from
human feedback (RLHF) for large language models (LLMs) by directly optimizing
human preferences without an explicit reward model. We find that during DPO
training, the reference model plays the role of a data weight adjuster.
However, the common practice of initializing the policy and reference models
identically in DPO can lead to inefficient data utilization and impose a
performance ceiling. Meanwhile, the lack of a reference model in Simple
Preference Optimization (SimPO) reduces training robustness and necessitates
stricter conditions to prevent catastrophic forgetting. In this work, we
propose Pre-DPO, a simple yet effective DPO-based training paradigm that
enhances preference optimization performance by leveraging a guiding reference
model. This reference model provides foresight into the optimal policy state
achievable through the training preference data, serving as a guiding mechanism
that adaptively assigns higher weights to samples more suitable for the model
and lower weights to those less suitable. Extensive experiments on AlpacaEval
2.0 and Arena-Hard v0.1 benchmarks demonstrate that Pre-DPO consistently
improves the performance of both DPO and SimPO, without relying on external
models or additional data.
Pre-DPO enhances preference optimization in RLHF for LLMs by using a guiding reference model to improve data utilization and performance.
Direct Preference Optimization DPO reinforcement learning from human feedback RLHF large language models LLMs reference model Simple Preference Optimization SimPO preference optimization AlpacaEval Arena-Hard