Safe RLHF: Safe Reinforcement Learning from Human Feedback
With the development of large language models (LLMs), striking a balance
between the performance and safety of AI systems has never been more critical.
However, the inherent tension between the objectives of helpfulness and
harmlessness presents a significant challenge during LLM training. To address
this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe
RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly
decouples human preferences regarding helpfulness and harmlessness, effectively
avoiding the crowdworkers' confusion about the tension and allowing us to train
separate reward and cost models. We formalize the safety concern of LLMs as an
optimization task of maximizing the reward function while satisfying specified
cost constraints. Leveraging the Lagrangian method to solve this constrained
problem, Safe RLHF dynamically adjusts the balance between the two objectives
during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we
demonstrate a superior ability to mitigate harmful responses while enhancing
model performance compared to existing value-aligned algorithms.
Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with
collected human preferences, significantly improving its helpfulness and
harmlessness according to human evaluations.
6532097ea70a88eb1e54c18e
Josef Dai
claimed_verified
2023-10-20T12:44:46.861Z
6426a75336ab74ff2bca0542
https://cdn-avatars.huggingface.co/v1/production/uploads/6426a75336ab74ff2bca0542/qL3QQofTr4yo8Cs58oYwK.png
Juntao Dai
user
calico-1226
6532097ea70a88eb1e54c18f
Xuehai Pan
admin_assigned
2023-10-20T08:54:09.602Z
64364e87fae2870051496e13
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t67EsNoRvRYXKwi0G59oa.jpeg
Xuehai Pan
user
XuehaiPan
6532097ea70a88eb1e54c190
Ruiyang Sun
admin_assigned
2023-10-20T08:54:28.486Z
63fca6b85d2bea4588bc9a94
/avatars/c8ae8c02d6d02af90678e8ee4122eafc.svg
Ruiyang Sun
user
RuiyangSun
6532097ea70a88eb1e54c191
Jiaming Ji
admin_assigned
2023-10-20T08:54:39.540Z
6429b13816d4d8293c97a949
https://cdn-avatars.huggingface.co/v1/production/uploads/6429b13816d4d8293c97a949/LNuQ1gvW2NMmWyIuEpzCA.jpeg
JiJiaming
user
jijiaming
6532097ea70a88eb1e54c192
Xinbo Xu
6532097ea70a88eb1e54c193
Mickel Liu
admin_assigned
2023-10-20T08:54:53.472Z
643f986a18afbc4d1f3e2456
/avatars/96a6e5f18386a9864dfdded8f25ec72d.svg
Mickel Liu
user
mickelliu
6532097ea70a88eb1e54c194
Yizhou Wang
6532097ea70a88eb1e54c195
Yaodong Yang
Safe RLHF: Safe Reinforcement Learning from Human Feedback
With the development of large language models (LLMs), striking a balance
between the performance and safety of AI systems has never been more critical.
However, the inherent tension between the objectives of helpfulness and
harmlessness presents a significant challenge during LLM training. To address
this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe
RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly
decouples human preferences regarding helpfulness and harmlessness, effectively
avoiding the crowdworkers' confusion about the tension and allowing us to train
separate reward and cost models. We formalize the safety concern of LLMs as an
optimization task of maximizing the reward function while satisfying specified
cost constraints. Leveraging the Lagrangian method to solve this constrained
problem, Safe RLHF dynamically adjusts the balance between the two objectives
during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we
demonstrate a superior ability to mitigate harmful responses while enhancing
model performance compared to existing value-aligned algorithms.
Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with
collected human preferences, significantly improving its helpfulness and
harmlessness according to human evaluations.
Safe RLHF, a novel algorithm for human value alignment in LLMs, improves performance and safety by decoupling human preferences and using a Lagrangian method to balance reward and cost constraints.
Safe RLHF human value alignment LLMs reinforcement learning from human feedback reward models cost models Lagrangian method fine-tuning human preferences helpfulness harmlessness