Aligning Large Language Models via Self-Steering Optimization
Automated alignment develops alignment systems with minimal human
intervention. The key to automated alignment lies in providing learnable and
accurate preference signals for preference learning without human annotation.
In this paper, we introduce Self-Steering Optimization (SSO), an algorithm
that autonomously generates high-quality preference signals based on predefined
principles during iterative training, eliminating the need for manual
annotation. SSO maintains the accuracy of signals by ensuring a consistent
gap between chosen and rejected responses while keeping them both on-policy to
suit the current policy model's learning capacity. SSO can benefit the online
and offline training of the policy model, as well as enhance the training of
reward models. We validate the effectiveness of SSO with two foundation
models, Qwen2 and Llama3.1, indicating that it provides accurate, on-policy
preference signals throughout iterative training. Without any manual annotation
or external models, SSO leads to significant performance improvements across
six subjective or objective benchmarks. Besides, the preference data generated
by SSO significantly enhanced the performance of the reward model on
Rewardbench. Our work presents a scalable approach to preference optimization,
paving the way for more efficient and effective automated alignment.
6718a271354948dfca7a1d1d
Hao Xiang
claimed_verified
2024-10-23T15:06:49.659Z
63f33d500be81bdc5d902356
/avatars/125812b9a86c3379b34ebfa8026f1a7f.svg
xianghao
user
xiangh
6718a271354948dfca7a1d1e
Bowen Yu
6718a271354948dfca7a1d1f
Hongyu Lin
admin_assigned
2024-10-23T08:00:36.142Z
6711c702f858a456b4b9f3a4
/avatars/178e9567c3111ab22717c3c0dd003a6a.svg
Hongyu  Lin
user
sanmusunrise
6718a271354948dfca7a1d20
Keming Lu
admin_assigned
2024-10-23T08:00:30.026Z
6453fa96ed6d7fede94408e0
/avatars/e8c9025ef24cec958c87a1008bb54fd7.svg
Keming Lu
user
keminglu
6718a271354948dfca7a1d21
Yaojie Lu
admin_assigned
2024-10-23T08:00:24.045Z
6216496a9b34d2fb49144599
https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
Yaojie Lu
user
luyaojie
6718a271354948dfca7a1d22
Xianpei Han
admin_assigned
2024-10-23T08:00:18.135Z
65e99a77e71555ed193609cf
/avatars/38ceb127883944677665da967d17dd18.svg
Xianpei Han
user
xphan
6718a271354948dfca7a1d23
Le Sun
6718a271354948dfca7a1d24
Jingren Zhou
admin_assigned
2024-10-23T08:00:02.877Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
6718a271354948dfca7a1d25
Junyang Lin
admin_assigned
2024-10-23T07:59:52.670Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
Aligning Large Language Models via Self-Steering Optimization
Automated alignment develops alignment systems with minimal human
intervention. The key to automated alignment lies in providing learnable and
accurate preference signals for preference learning without human annotation.
In this paper, we introduce Self-Steering Optimization (SSO), an algorithm
that autonomously generates high-quality preference signals based on predefined
principles during iterative training, eliminating the need for manual
annotation. SSO maintains the accuracy of signals by ensuring a consistent
gap between chosen and rejected responses while keeping them both on-policy to
suit the current policy model's learning capacity. SSO can benefit the online
and offline training of the policy model, as well as enhance the training of
reward models. We validate the effectiveness of SSO with two foundation
models, Qwen2 and Llama3.1, indicating that it provides accurate, on-policy
preference signals throughout iterative training. Without any manual annotation
or external models, SSO leads to significant performance improvements across
six subjective or objective benchmarks. Besides, the preference data generated
by SSO significantly enhanced the performance of the reward model on
Rewardbench. Our work presents a scalable approach to preference optimization,
paving the way for more efficient and effective automated alignment.
Self-Steering Optimization autonomously generates accurate preference signals without manual annotation, improving policy and reward model performance.
Self-Steering Optimization SSO preference learning preference signals iterative training policy model reward model