Finding the Sweet Spot: Preference Data Construction for Scaling
  Preference Optimization
Iterative data generation and model retraining are widely used to align large
language models (LLMs). It typically involves a policy model to generate
on-policy responses and a reward model to guide training data selection. Direct
Preference Optimization (DPO) further enhances this process by constructing
preference pairs of chosen and rejected responses. In this work, we aim to
scale up the number of on-policy samples via repeated random sampling to
improve alignment performance. Conventional practice selects the sample with
the highest reward as chosen and the lowest as rejected for DPO. However, our
experiments reveal that this strategy leads to a decline in performance
as the sample size increases. To address this, we investigate preference data
construction through the lens of underlying normal distribution of sample
rewards. We categorize the reward space into seven representative points and
systematically explore all 21 (C_7^2) pairwise combinations. Through
evaluations on four models using AlpacaEval 2, we find that selecting the
rejected response at reward position mu - 2sigma rather than the minimum
reward, is crucial for optimal performance. We finally introduce a scalable
preference data construction strategy that consistently enhances model
performance as the sample scale increases.
67bf243823f222a2cc2858d0
Yao Xiao
67bf243823f222a2cc2858d1
Hai Ye
67bf243823f222a2cc2858d2
Linyao Chen
67bf243823f222a2cc2858d3
Hwee Tou Ng
67bf243823f222a2cc2858d4
Lidong Bing
claimed_verified
2025-07-22T07:55:03.147Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
67bf243823f222a2cc2858d5
Xiaoli Li
67bf243823f222a2cc2858d6
Roy Ka-wei Lee
Finding the Sweet Spot: Preference Data Construction for Scaling
  Preference Optimization
Iterative data generation and model retraining are widely used to align large
language models (LLMs). It typically involves a policy model to generate
on-policy responses and a reward model to guide training data selection. Direct
Preference Optimization (DPO) further enhances this process by constructing
preference pairs of chosen and rejected responses. In this work, we aim to
scale up the number of on-policy samples via repeated random sampling to
improve alignment performance. Conventional practice selects the sample with
the highest reward as chosen and the lowest as rejected for DPO. However, our
experiments reveal that this strategy leads to a decline in performance
as the sample size increases. To address this, we investigate preference data
construction through the lens of underlying normal distribution of sample
rewards. We categorize the reward space into seven representative points and
systematically explore all 21 (C_7^2) pairwise combinations. Through
evaluations on four models using AlpacaEval 2, we find that selecting the
rejected response at reward position mu - 2sigma rather than the minimum
reward, is crucial for optimal performance. We finally introduce a scalable
preference data construction strategy that consistently enhances model
performance as the sample scale increases.
A scalable preference data construction strategy improves the alignment of large language models by selecting optimal preference pairs based on the normal distribution of sample rewards.
large language models LLMs policy model reward model Direct Preference Optimization DPO on-policy samples preference pairs reward space AlpacaEval normal distribution sample rewards