Direct Preference Optimization Using Sparse Feature-Level Constraints
The alignment of large language models (LLMs) with human preferences remains
a key challenge. While post-training techniques like Reinforcement Learning
from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have
achieved notable success, they often introduce computational inefficiencies and
training instability. In this paper, we propose Feature-level constrained
Preference Optimization (FPO), a novel method designed to simplify the
alignment process while ensuring stability. FPO leverages pre-trained Sparse
Autoencoders (SAEs) and introduces feature-level constraints, allowing for
efficient, sparsity-enforced alignment. Our approach enjoys efficiency by using
sparse features activated in a well-trained sparse autoencoder and the quality
of sequential KL divergence by using the feature-level offline reference.
Experimental results on benchmark datasets demonstrate that FPO achieves a
5.08% absolute improvement in win rate with much lower computational cost
compared to state-of-the-art baselines, making it a promising solution for
efficient and controllable LLM alignments.
673575eee12f354dfc1ea6f8
Qingyu Yin
admin_assigned
2024-11-15T14:16:41.130Z
6453cb22908e259483c0a061
https://cdn-avatars.huggingface.co/v1/production/uploads/6453cb22908e259483c0a061/hMgdwZUsUbgquGalzPGzV.jpeg
Qingyu_Yin
user
MikaStars39
673575eee12f354dfc1ea6f9
Chak Tou Leong
claimed_verified
2024-11-15T09:22:14.403Z
631326d6289cf15634c52369
https://cdn-avatars.huggingface.co/v1/production/uploads/631326d6289cf15634c52369/lmPWGHLsQ36H556cqcXjT.jpeg
Cooper Leong
user
cooperleong00
673575eee12f354dfc1ea6fa
Hongbo Zhang
admin_assigned
2024-11-15T14:17:14.035Z
660687a8cd1b5614c65e257b
/avatars/64ef0ff1d5bff7ff5aadd9070abbff96.svg
zhanghongbo
user
loewen-hob
673575eee12f354dfc1ea6fb
Minjun Zhu
admin_assigned
2024-11-15T14:17:27.675Z
622ee9f3165ba2c1bcbc7706
https://cdn-avatars.huggingface.co/v1/production/uploads/622ee9f3165ba2c1bcbc7706/4BIp_juKHqXgnctnNIFJf.jpeg
MinjunZhu
user
MinjunZhu
673575eee12f354dfc1ea6fc
Hanqi Yan
admin_assigned
2024-11-15T14:17:20.271Z
654d3672f0f195070248d80b
/avatars/8f54607d3189690b90e4fe5b6fba94f1.svg
hanqi yan
user
hanqiyan
673575eee12f354dfc1ea6fd
Qiang Zhang
673575eee12f354dfc1ea6fe
Yulan He
673575eee12f354dfc1ea6ff
Wenjie Li
admin_assigned
2024-11-15T14:18:04.907Z
66a3710a4ee2a4c936315a5a
/avatars/ef8da8fb1031695d77d34a5d365aa177.svg
Li
user
WenjieLi
673575eee12f354dfc1ea700
Jun Wang
673575eee12f354dfc1ea701
Yue Zhang
673575eee12f354dfc1ea702
Linyi Yang
admin_assigned
2024-11-15T14:18:24.971Z
64895683f534abe18eec264b
/avatars/73cc9e6db6db86793787750776b57c63.svg
Linyi Yang
user
linyiyang2023
Direct Preference Optimization Using Sparse Feature-Level Constraints
The alignment of large language models (LLMs) with human preferences remains
a key challenge. While post-training techniques like Reinforcement Learning
from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have
achieved notable success, they often introduce computational inefficiencies and
training instability. In this paper, we propose Feature-level constrained
Preference Optimization (FPO), a novel method designed to simplify the
alignment process while ensuring stability. FPO leverages pre-trained Sparse
Autoencoders (SAEs) and introduces feature-level constraints, allowing for
efficient, sparsity-enforced alignment. Our approach enjoys efficiency by using
sparse features activated in a well-trained sparse autoencoder and the quality
of sequential KL divergence by using the feature-level offline reference.
Experimental results on benchmark datasets demonstrate that FPO achieves a
5.08% absolute improvement in win rate with much lower computational cost
compared to state-of-the-art baselines, making it a promising solution for
efficient and controllable LLM alignments.
A novel method, Feature-level constrained Preference Optimization (FPO), improves the alignment of large language models with human preferences by using sparse features from Sparse Autoencoders and feature-level offline reference, achieving better efficiency and win rate.
Large language models Reinforcement Learning from Human Feedback (RLHF) Direct Preference Optimization (DPO) Feature-level constrained Preference Optimization (FPO) Sparse Autoencoders (SAEs) feature-level constraints sequential KL divergence feature-level offline reference