Modulated Intervention Preference Optimization (MIPO): Keep the Easy,
  Refine the Difficult
Preference optimization methods typically begin training with a well-trained
SFT model as a reference model. In RLHF and DPO, a regularization term is used
during the preference optimization process to prevent the policy model from
deviating too far from the reference model's distribution, thereby avoiding the
generation of anomalous responses. When the reference model is already
well-aligned with the given data or only requires slight adjustments, this
approach can produce a well-aligned model. However, if the reference model is
not aligned with the given data and requires significant deviation from its
current state, a regularization term may actually hinder the model alignment.
In this study, we propose Modulated Intervention Preference
Optimization (MIPO) to address this issue. MIPO modulates the degree of
intervention from the reference model based on how well the given data is
aligned with it. If the data is well-aligned, the intervention is increased to
prevent the policy model from diverging significantly from reference model.
Conversely, if the alignment is poor, the interference is reduced to facilitate
more extensive training. We compare the performance of MIPO and DPO using
Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench. The experimental
results demonstrate that MIPO consistently outperforms DPO across various
evaluation scenarios.
66fa2848f50b1d60294265e8
Cheolhun Jang
claimed_verified
2024-09-30T08:06:09.218Z
64757d6322b4dec4524c924c
/avatars/51c1a93f44375e2604d4c23ca66600ad.svg
Cheolhun Jang
user
fetong
Modulated Intervention Preference Optimization (MIPO): Keep the Easy,
  Refine the Difficult
Preference optimization methods typically begin training with a well-trained
SFT model as a reference model. In RLHF and DPO, a regularization term is used
during the preference optimization process to prevent the policy model from
deviating too far from the reference model's distribution, thereby avoiding the
generation of anomalous responses. When the reference model is already
well-aligned with the given data or only requires slight adjustments, this
approach can produce a well-aligned model. However, if the reference model is
not aligned with the given data and requires significant deviation from its
current state, a regularization term may actually hinder the model alignment.
In this study, we propose Modulated Intervention Preference
Optimization (MIPO) to address this issue. MIPO modulates the degree of
intervention from the reference model based on how well the given data is
aligned with it. If the data is well-aligned, the intervention is increased to
prevent the policy model from diverging significantly from reference model.
Conversely, if the alignment is poor, the interference is reduced to facilitate
more extensive training. We compare the performance of MIPO and DPO using
Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench. The experimental
results demonstrate that MIPO consistently outperforms DPO across various
evaluation scenarios.
Modulated Intervention Preference Optimization (MIPO) adapts the degree of intervention from a reference model based on data alignment, improving performance over traditional methods like DPO in preference optimization tasks.
Modulated Intervention Preference Optimization MIPO preference optimization RLHF DPO regularization term policy model reference model well-aligned data alignment divergence