SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in
  Language Model Preference Learning
Aligning language models with human preferences relies on pairwise preference
datasets. While some studies suggest that on-policy data consistently
outperforms off -policy data for preference learning, others indicate that the
advantages of on-policy data may be task-dependent, highlighting the need for a
systematic exploration of their interplay.
  In this work, we show that on-policy and off-policy data offer complementary
strengths in preference optimization: on-policy data is particularly effective
for reasoning tasks like math and coding, while off-policy data performs better
on open-ended tasks such as creative writing and making personal
recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach
to combine the complementary strengths of on-policy and off-policy preference
learning by simply mixing these two data sources. Our empirical results across
diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves
language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO
and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it
outperforms prior approaches that are much more complex in combining on- and
off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.
681a1efda513c2226dff667c
Tianjian Li
admin_assigned
2025-05-14T12:57:55.901Z
62bc72d464b7a623f3a242e9
/avatars/db79848dc78a5c4927e27de4426c3b47.svg
Tianjian Li
user
dogtooth
681a1efda513c2226dff667d
Daniel Khashabi
admin_assigned
2025-05-14T12:58:03.184Z
5f6540c65e78cc6b0ed3199d
/avatars/0280d4df417855965a0964d22766c012.svg
Daniel Khashabi
user
danyaljj
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in
  Language Model Preference Learning
Aligning language models with human preferences relies on pairwise preference
datasets. While some studies suggest that on-policy data consistently
outperforms off -policy data for preference learning, others indicate that the
advantages of on-policy data may be task-dependent, highlighting the need for a
systematic exploration of their interplay.
  In this work, we show that on-policy and off-policy data offer complementary
strengths in preference optimization: on-policy data is particularly effective
for reasoning tasks like math and coding, while off-policy data performs better
on open-ended tasks such as creative writing and making personal
recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach
to combine the complementary strengths of on-policy and off-policy preference
learning by simply mixing these two data sources. Our empirical results across
diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves
language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO
and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it
outperforms prior approaches that are much more complex in combining on- and
off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.
A combination of on-policy and off-policy data enhances language model alignment across various tasks, outperforming complex integration methods.
pairwise preference datasets preference learning on-policy data off-policy data preference optimization SIMPLEMIX reasoning tasks open-ended tasks language model alignment DPO Alpaca Eval 2.0 HyPO DPO-Mix-P