Learning to Optimize Multi-Objective Alignment Through Dynamic Reward
  Weighting
Prior works in multi-objective reinforcement learning typically use linear
reward scalarization with fixed weights, which provably fail to capture
non-convex Pareto fronts and thus yield suboptimal results. This limitation
becomes especially critical in online preference alignment for large language
models. Here, stochastic trajectories generated by parameterized policies
create highly non-linear and non-convex mappings from parameters to objectives
that no single static weighting scheme can find optimal trade-offs. We address
this limitation by introducing dynamic reward weighting, which adaptively
adjusts reward weights during the online reinforcement learning process. Unlike
existing approaches that rely on fixed-weight interpolation, our dynamic
weighting continuously balances and prioritizes objectives in training,
facilitating effective exploration of Pareto fronts in objective space. We
introduce two approaches of increasing sophistication and generalizability: (1)
hypervolume-guided weight adaptation and (2) gradient-based weight
optimization, offering a versatile toolkit for online multi-objective
alignment. Our extensive experiments demonstrate their compatibility with
commonly used online reinforcement learning algorithms (including GRPO,
REINFORCE, and RLOO), effectiveness across multiple mathematical reasoning
datasets, and applicability to different model families, consistently achieving
Pareto dominant solutions with fewer training steps than fixed-weight linear
scalarization baselines.
68c8c651733e345e52ac1e5b
Yining Lu
claimed_verified
2025-09-16T09:42:41.060Z
642f742270daaa6e7209a2c8
/avatars/746fb840c220329f69a17905ea519322.svg
Yining Lu
user
ylu610
68c8c651733e345e52ac1e5c
Zilong Wang
68c8c651733e345e52ac1e5d
Shiyang Li
68c8c651733e345e52ac1e5e
Xin Liu
68c8c651733e345e52ac1e5f
Changlong Yu
68c8c651733e345e52ac1e60
Qingyu Yin
68c8c651733e345e52ac1e61
Zhan Shi
68c8c651733e345e52ac1e62
Zixuan Zhang
68c8c651733e345e52ac1e63
Meng Jiang
Learning to Optimize Multi-Objective Alignment Through Dynamic Reward
  Weighting
Prior works in multi-objective reinforcement learning typically use linear
reward scalarization with fixed weights, which provably fail to capture
non-convex Pareto fronts and thus yield suboptimal results. This limitation
becomes especially critical in online preference alignment for large language
models. Here, stochastic trajectories generated by parameterized policies
create highly non-linear and non-convex mappings from parameters to objectives
that no single static weighting scheme can find optimal trade-offs. We address
this limitation by introducing dynamic reward weighting, which adaptively
adjusts reward weights during the online reinforcement learning process. Unlike
existing approaches that rely on fixed-weight interpolation, our dynamic
weighting continuously balances and prioritizes objectives in training,
facilitating effective exploration of Pareto fronts in objective space. We
introduce two approaches of increasing sophistication and generalizability: (1)
hypervolume-guided weight adaptation and (2) gradient-based weight
optimization, offering a versatile toolkit for online multi-objective
alignment. Our extensive experiments demonstrate their compatibility with
commonly used online reinforcement learning algorithms (including GRPO,
REINFORCE, and RLOO), effectiveness across multiple mathematical reasoning
datasets, and applicability to different model families, consistently achieving
Pareto dominant solutions with fewer training steps than fixed-weight linear
scalarization baselines.
Dynamic reward weighting in multi-objective reinforcement learning adaptively adjusts weights during training to explore Pareto fronts effectively, outperforming fixed-weight scalarization methods.
multi-objective reinforcement learning linear reward scalarization non-convex Pareto fronts parameterized policies dynamic reward weighting hypervolume-guided weight adaptation gradient-based weight optimization GRPO REINFORCE RLOO Pareto dominant solutions