Alignment through Meta-Weighted Online Sampling: Bridging the Gap
  between Data Generation and Preference Optimization
Preference optimization is crucial for aligning large language models (LLMs)
with human values and intentions. A significant challenge in this process is
the distribution mismatch between pre-collected offline preference data and the
evolving model policy. Existing methods attempt to reduce this gap using static
heuristics or decoupled online sampling strategies, but they often fail to
adapt to the model's dynamic learning state. To bridge this gap, we propose
Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework
that dynamically couples data generation with model training. MetaAPO employs a
lightweight meta-learner, as an "alignment gap estimator", to evaluate the
potential benefits of on-policy sampling in relation to offline data. This
guides targeted online generation and assigns sample-wise meta-weights to the
optimization objective, dynamically balancing the quality and distribution of
online and offline data. Experiments on AlpacaEval 2, Arena-Hard and MT-Bench
demonstrate that MetaAPO consistently outperforms existing preference
optimization approaches across various settings, while reducing 42% in online
annotation costs.
68db3818d2bf1f4b15ec7290
Junming Yang
claimed_verified
2025-09-30T06:34:49.878Z
653f1d243bd61358055ad51d
/avatars/698c03b9a4bb69659d2ed594626e3895.svg
junmingyang
user
jmyang
68db3818d2bf1f4b15ec7291
Ning Xu
68db3818d2bf1f4b15ec7292
Biao Liu
68db3818d2bf1f4b15ec7293
Shiqi Qiao
68db3818d2bf1f4b15ec7294
Xin Geng
Alignment through Meta-Weighted Online Sampling: Bridging the Gap
  between Data Generation and Preference Optimization
Preference optimization is crucial for aligning large language models (LLMs)
with human values and intentions. A significant challenge in this process is
the distribution mismatch between pre-collected offline preference data and the
evolving model policy. Existing methods attempt to reduce this gap using static
heuristics or decoupled online sampling strategies, but they often fail to
adapt to the model's dynamic learning state. To bridge this gap, we propose
Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework
that dynamically couples data generation with model training. MetaAPO employs a
lightweight meta-learner, as an "alignment gap estimator", to evaluate the
potential benefits of on-policy sampling in relation to offline data. This
guides targeted online generation and assigns sample-wise meta-weights to the
optimization objective, dynamically balancing the quality and distribution of
online and offline data. Experiments on AlpacaEval 2, Arena-Hard and MT-Bench
demonstrate that MetaAPO consistently outperforms existing preference
optimization approaches across various settings, while reducing 42% in online
annotation costs.
Meta-Weighted Adaptive Preference Optimization (MetaAPO) dynamically balances online and offline data to align large language models with human preferences, outperforming existing methods and reducing annotation costs.
Meta-Weighted Adaptive Preference Optimization MetaAPO meta-learner alignment gap estimator on-policy sampling AlpacaEval 2 Arena-Hard MT-Bench