RePO: ReLU-based Preference Optimization
Aligning large language models (LLMs) with human preferences is critical for
real-world deployment, yet existing methods like RLHF face computational and
stability challenges. While DPO establishes an offline paradigm with single
hyperparameter beta, subsequent methods like SimPO reintroduce complexity
through dual parameters (beta, gamma). We propose {ReLU-based Preference
Optimization (RePO)}, a streamlined algorithm that eliminates beta via two
advances: (1) retaining SimPO's reference-free margins but removing beta
through gradient analysis, and (2) adopting a ReLU-based max-margin loss that
naturally filters trivial pairs. Theoretically, RePO is characterized as
SimPO's limiting case (beta to infty), where the logistic weighting
collapses to binary thresholding, forming a convex envelope of the 0-1 loss.
Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO
and SimPO across multiple base models, requiring only one hyperparameter to
tune.
67cff321f2b1fe815dce3722
Junkang Wu
admin_assigned
2025-03-11T14:16:45.903Z
664764e5d834283e7ff96d37
https://cdn-avatars.huggingface.co/v1/production/uploads/664764e5d834283e7ff96d37/JBi3dvAUKqqGf_DjVORK7.jpeg
Junkang Wu
user
junkang0909
67cff321f2b1fe815dce3723
Kexin Huang
admin_assigned
2025-03-11T14:16:08.313Z
631fc126788f34547a9d7ee1
https://cdn-avatars.huggingface.co/v1/production/uploads/1674202999503-631fc126788f34547a9d7ee1.png
Kexin Huang
user
kexinhuang
67cff321f2b1fe815dce3724
Xue Wang
67cff321f2b1fe815dce3725
Jinyang Gao
67cff321f2b1fe815dce3726
Bolin Ding
67cff321f2b1fe815dce3727
Jiancan Wu
67cff321f2b1fe815dce3728
Xiangnan He
67cff321f2b1fe815dce3729
Xiang Wang
extracted_pending
2025-03-11T08:24:02.839Z
65fca775fa59bdf4737b1a84
/avatars/a161b510bde8f57e7686cbb0b4aa6a52.svg
Xiang Wang
user
xiangwang1223
RePO: ReLU-based Preference Optimization
Aligning large language models (LLMs) with human preferences is critical for
real-world deployment, yet existing methods like RLHF face computational and
stability challenges. While DPO establishes an offline paradigm with single
hyperparameter beta, subsequent methods like SimPO reintroduce complexity
through dual parameters (beta, gamma). We propose {ReLU-based Preference
Optimization (RePO)}, a streamlined algorithm that eliminates beta via two
advances: (1) retaining SimPO's reference-free margins but removing beta
through gradient analysis, and (2) adopting a ReLU-based max-margin loss that
naturally filters trivial pairs. Theoretically, RePO is characterized as
SimPO's limiting case (beta to infty), where the logistic weighting
collapses to binary thresholding, forming a convex envelope of the 0-1 loss.
Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO
and SimPO across multiple base models, requiring only one hyperparameter to
tune.
RePO is a streamlined preference optimization algorithm for aligning large language models with human preferences, performing better than DPO and SimPO while using only one hyperparameter.
RLHF DPO SimPO ReLU-based Preference Optimization reference-free margins max-margin loss ReLU convex envelope 0-1 loss hyperparameter AlpacaEval Arena-Hard