Margin Adaptive DPO: Leveraging Reward Model for Granular Control in
  Preference Optimization
Direct Preference Optimization (DPO) has emerged as a simple and effective
method for aligning large language models. However, its reliance on a fixed
temperature parameter leads to suboptimal training on diverse preference data,
causing overfitting on easy examples and under-learning from informative ones.
Recent methods have emerged to counter this. While IPO addresses general
overfitting, its uniform regularization can be overly conservative. The more
targeted approach of beta-DPO suffers from its own limitations: its
batch-level adaptation applies a single, compromised temperature to
mixed-margin pairs, its linear update rule can produce unstable negative
beta values, and its filtering mechanism discards potentially useful
training signals. In this work, we introduce Margin-Adaptive Direct Preference
Optimization (MADPO), a method that provides a stable, data-preserving, and
instance-level solution. MADPO employs a practical two-step approach: it first
trains a reward model to estimate preference margins and then uses these
margins to apply a continuous, adaptive weight to the DPO loss for each
individual training sample. This re-weighting scheme creates an effective
target margin that is amplified for hard pairs and dampened for easy pairs,
allowing for granular control over the learning signal. We provide a
comprehensive theoretical analysis, proving that MADPO has a well-behaved
optimization landscape and is robust to reward model estimation errors. We
validate our theory with experiments on a sentiment generation task, where
MADPO consistently and significantly outperforms strong baselines across
datasets of varying quality. It achieves performance gains of up to +33.3\% on
High Quality data and +10.5\% on Low Quality data over the next-best method.
Our results establish MADPO as a more robust and principled approach to
preference alignment.
68e5b6f9975ac4c405ef2003
Hyung Gyu Rho
claimed_verified
2025-10-08T08:00:52.339Z
6642dafed48363a46ddb69ed
/avatars/111fb26ea38fb2e6f8470f7ed513d48d.svg
hyung gyu rho
user
sirano1004
Margin Adaptive DPO: Leveraging Reward Model for Granular Control in
  Preference Optimization
Direct Preference Optimization (DPO) has emerged as a simple and effective
method for aligning large language models. However, its reliance on a fixed
temperature parameter leads to suboptimal training on diverse preference data,
causing overfitting on easy examples and under-learning from informative ones.
Recent methods have emerged to counter this. While IPO addresses general
overfitting, its uniform regularization can be overly conservative. The more
targeted approach of beta-DPO suffers from its own limitations: its
batch-level adaptation applies a single, compromised temperature to
mixed-margin pairs, its linear update rule can produce unstable negative
beta values, and its filtering mechanism discards potentially useful
training signals. In this work, we introduce Margin-Adaptive Direct Preference
Optimization (MADPO), a method that provides a stable, data-preserving, and
instance-level solution. MADPO employs a practical two-step approach: it first
trains a reward model to estimate preference margins and then uses these
margins to apply a continuous, adaptive weight to the DPO loss for each
individual training sample. This re-weighting scheme creates an effective
target margin that is amplified for hard pairs and dampened for easy pairs,
allowing for granular control over the learning signal. We provide a
comprehensive theoretical analysis, proving that MADPO has a well-behaved
optimization landscape and is robust to reward model estimation errors. We
validate our theory with experiments on a sentiment generation task, where
MADPO consistently and significantly outperforms strong baselines across
datasets of varying quality. It achieves performance gains of up to +33.3\% on
High Quality data and +10.5\% on Low Quality data over the next-best method.
Our results establish MADPO as a more robust and principled approach to
preference alignment.
MADPO, a margin-adaptive method, enhances preference alignment in large language models by providing instance-level adaptive weighting to the DPO loss, improving performance across datasets.
Direct Preference Optimization DPO IPO β-DPO reward model preference margins DPO loss optimization landscape reward model estimation errors sentiment generation task