SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
In Large Language Model (LLM) development, Reinforcement Learning from Human
Feedback (RLHF) is crucial for aligning models with human values and
preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence
between the current policy and a frozen initial policy as a reference, which is
added as a penalty in policy optimization algorithms like Proximal Policy
Optimization (PPO). While this constraint prevents models from deviating too
far from the initial checkpoint, it limits exploration of the reward landscape,
reducing the model's ability to discover higher-quality solutions. As a result,
policy optimization is often trapped in a narrow region of the parameter space,
leading to suboptimal alignment and performance. This paper presents SALSA
(Soup-based Alignment Learning for Stronger Adaptation), a novel approach
designed to overcome these limitations by creating a more flexible and better
located reference model through weight-space averaging of two independent
supervised fine-tuned (SFT) models. This model soup allows for larger deviation
in KL divergence and exploring a promising region of the solution space without
sacrificing stability. By leveraging this more robust reference model, SALSA
fosters better exploration, achieving higher rewards and improving model
robustness, out-of-distribution generalization, and performance. We validate
the effectiveness of SALSA through extensive experiments on popular open models
(Llama2-7B, Mistral-7B, and Gemma-2B) across various benchmarks (MT-Bench,
Arena-Hard, UltraFeedback), where it consistently surpasses PPO by fostering
deeper exploration and achieving superior alignment in LLMs.
672a8daf673a4e2d70053a50
Atoosa Chegini
672a8daf673a4e2d70053a51
Hamid Kazemi
claimed_verified
2024-11-15T09:37:35.271Z
64f1f9612820a6f1b9e19edd
https://cdn-avatars.huggingface.co/v1/production/uploads/64f1f9612820a6f1b9e19edd/hSmhLsczKR1t5jxHwEaE_.jpeg
Hamid Kazemi
user
seyedhamidreza
672a8daf673a4e2d70053a52
Iman Mirzadeh
672a8daf673a4e2d70053a53
Dong Yin
672a8daf673a4e2d70053a54
Maxwell Horton
672a8daf673a4e2d70053a55
Moin Nabi
672a8daf673a4e2d70053a56
Mehrdad Farajtabar
672a8daf673a4e2d70053a57
Keivan Alizadeh
SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
In Large Language Model (LLM) development, Reinforcement Learning from Human
Feedback (RLHF) is crucial for aligning models with human values and
preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence
between the current policy and a frozen initial policy as a reference, which is
added as a penalty in policy optimization algorithms like Proximal Policy
Optimization (PPO). While this constraint prevents models from deviating too
far from the initial checkpoint, it limits exploration of the reward landscape,
reducing the model's ability to discover higher-quality solutions. As a result,
policy optimization is often trapped in a narrow region of the parameter space,
leading to suboptimal alignment and performance. This paper presents SALSA
(Soup-based Alignment Learning for Stronger Adaptation), a novel approach
designed to overcome these limitations by creating a more flexible and better
located reference model through weight-space averaging of two independent
supervised fine-tuned (SFT) models. This model soup allows for larger deviation
in KL divergence and exploring a promising region of the solution space without
sacrificing stability. By leveraging this more robust reference model, SALSA
fosters better exploration, achieving higher rewards and improving model
robustness, out-of-distribution generalization, and performance. We validate
the effectiveness of SALSA through extensive experiments on popular open models
(Llama2-7B, Mistral-7B, and Gemma-2B) across various benchmarks (MT-Bench,
Arena-Hard, UltraFeedback), where it consistently surpasses PPO by fostering
deeper exploration and achieving superior alignment in LLMs.
SALSA, a novel RLHF method using weight-space averaging of supervised fine-tuned models, improves LLM alignment and performance by allowing larger KL divergence and better exploration of the solution space.
Reinforcement Learning from Human Feedback (RLHF) Kullback-Leibler (KL) divergence Proximal Policy Optimization (PPO) policy optimization model soup supervised fine-tuned (SFT) models weight-space averaging solution space exploration model robustness out-of-distribution generalization MT-Bench Arena-Hard UltraFeedback