Learning to Align, Aligning to Learn: A Unified Approach for
  Self-Optimized Alignment
Alignment methodologies have emerged as a critical pathway for enhancing
language model alignment capabilities. While SFT (supervised fine-tuning)
accelerates convergence through direct token-level loss intervention, its
efficacy is constrained by offline policy trajectory. In contrast,
RL(reinforcement learning) facilitates exploratory policy optimization, but
suffers from low sample efficiency and stringent dependency on high-quality
base models. To address these dual challenges, we propose GRAO (Group Relative
Alignment Optimization), a unified framework that synergizes the respective
strengths of SFT and RL through three key innovations: 1) A multi-sample
generation strategy enabling comparative quality assessment via reward
feedback; 2) A novel Group Direct Alignment Loss formulation leveraging
intra-group relative advantage weighting; 3) Reference-aware parameter updates
guided by pairwise preference dynamics. Our theoretical analysis establishes
GRAO's convergence guarantees and sample efficiency advantages over
conventional approaches. Comprehensive evaluations across complex human
alignment tasks demonstrate GRAO's superior performance, achieving
57.70\%,17.65\% 7.95\% and 5.18\% relative improvements over SFT, DPO, PPO and
GRPO baselines respectively. This work provides both a theoretically grounded
alignment framework and empirical evidence for efficient capability evolution
in language models.
689d9cfbb083e610d741eaf1
Haowen Wang
claimed_verified
2025-08-14T13:37:04.723Z
64b509197da6a1dca8b20bb7
/avatars/0d3304b017b8ee735a9686123cec0b99.svg
Haowen Wang
user
whw199833
689d9cfbb083e610d741eaf2
Yun Yue
689d9cfbb083e610d741eaf3
Zhiling Ye
claimed_verified
2025-08-14T13:37:07.534Z
63d7bf58db69ae9b9beaeea3
https://cdn-avatars.huggingface.co/v1/production/uploads/63d7bf58db69ae9b9beaeea3/xO1nkkmhJGIwRuk6yQDPi.png
Ye Zhiling
user
yzlnew
689d9cfbb083e610d741eaf4
Shuowen Zhang
689d9cfbb083e610d741eaf5
Lei Fan
689d9cfbb083e610d741eaf6
Jiaxin Liang
689d9cfbb083e610d741eaf7
Jiadi Jiang
689d9cfbb083e610d741eaf8
Cheng Wei
689d9cfbb083e610d741eaf9
Jingyuan Deng
689d9cfbb083e610d741eafa
Xudong Han
689d9cfbb083e610d741eafb
Ji Li
689d9cfbb083e610d741eafc
Chunxiao Guo
689d9cfbb083e610d741eafd
Peng Wei
689d9cfbb083e610d741eafe
Jian Wang
689d9cfbb083e610d741eaff
Jinjie Gu
Learning to Align, Aligning to Learn: A Unified Approach for
  Self-Optimized Alignment
Alignment methodologies have emerged as a critical pathway for enhancing
language model alignment capabilities. While SFT (supervised fine-tuning)
accelerates convergence through direct token-level loss intervention, its
efficacy is constrained by offline policy trajectory. In contrast,
RL(reinforcement learning) facilitates exploratory policy optimization, but
suffers from low sample efficiency and stringent dependency on high-quality
base models. To address these dual challenges, we propose GRAO (Group Relative
Alignment Optimization), a unified framework that synergizes the respective
strengths of SFT and RL through three key innovations: 1) A multi-sample
generation strategy enabling comparative quality assessment via reward
feedback; 2) A novel Group Direct Alignment Loss formulation leveraging
intra-group relative advantage weighting; 3) Reference-aware parameter updates
guided by pairwise preference dynamics. Our theoretical analysis establishes
GRAO's convergence guarantees and sample efficiency advantages over
conventional approaches. Comprehensive evaluations across complex human
alignment tasks demonstrate GRAO's superior performance, achieving
57.70\%,17.65\% 7.95\% and 5.18\% relative improvements over SFT, DPO, PPO and
GRPO baselines respectively. This work provides both a theoretically grounded
alignment framework and empirical evidence for efficient capability evolution
in language models.
GRAO, a unified framework combining SFT and RL, enhances language model alignment through multi-sample generation, Group Direct Alignment Loss, and reference-aware parameter updates, demonstrating superior performance across human alignment tasks.
supervised fine-tuning reinforcement learning GRAO multi-sample generation Group Direct Alignment Loss reference-aware parameter updates convergence guarantees sample efficiency