Course-Correction: Safety Alignment Using Synthetic Preferences
The risk of harmful content generated by large language models (LLMs) becomes
a critical concern. This paper presents a systematic study on assessing and
improving LLMs' capability to perform the task of course-correction,
\ie, the model can steer away from generating harmful content autonomously. To
start with, we introduce the C^2-Eval benchmark for quantitative
assessment and analyze 10 popular LLMs, revealing varying proficiency of
current safety-tuned LLMs in course-correction. To improve, we propose
fine-tuning LLMs with preference learning, emphasizing the preference for
timely course-correction. Using an automated pipeline, we create
C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
teach models the concept of timely course-correction through data-driven
preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
Qwen2 7B, show that our method effectively enhances course-correction
skills without affecting general performance. Additionally, it effectively
improves LLMs' safety, particularly in resisting jailbreak attacks.
66a378e49b6c8a5bd7ddf2b4
Rongwu Xu
claimed_verified
2024-07-26T10:26:39.119Z
64ed86a5b6d4b4b02a9190cf
/avatars/8b92a24566ecb3489a3adac0586d4dd6.svg
Rongwu Xu
user
pillowsofwind
66a378e49b6c8a5bd7ddf2b5
Yishuo Cai
66a378e49b6c8a5bd7ddf2b6
Zhenhong Zhou
66a378e49b6c8a5bd7ddf2b7
Renjie Gu
admin_assigned
2024-07-26T12:20:33.035Z
668657b4c9ffbedc86831006
/avatars/0d766ccde2e9890c1d61cc5d04e1d692.svg
Renjie Gu
user
csuJC
66a378e49b6c8a5bd7ddf2b8
Haiqin Weng
admin_assigned
2024-07-26T12:20:26.728Z
63fef648c0ec83fda43da041
/avatars/a24ce8fc339f9b36237d241f741472ea.svg
Haiqin Weng
user
haiqin
66a378e49b6c8a5bd7ddf2b9
Yan Liu
66a378e49b6c8a5bd7ddf2ba
Tianwei Zhang
66a378e49b6c8a5bd7ddf2bb
Wei Xu
66a378e49b6c8a5bd7ddf2bc
Han Qiu
Course-Correction: Safety Alignment Using Synthetic Preferences
The risk of harmful content generated by large language models (LLMs) becomes
a critical concern. This paper presents a systematic study on assessing and
improving LLMs' capability to perform the task of course-correction,
\ie, the model can steer away from generating harmful content autonomously. To
start with, we introduce the C^2-Eval benchmark for quantitative
assessment and analyze 10 popular LLMs, revealing varying proficiency of
current safety-tuned LLMs in course-correction. To improve, we propose
fine-tuning LLMs with preference learning, emphasizing the preference for
timely course-correction. Using an automated pipeline, we create
C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
teach models the concept of timely course-correction through data-driven
preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
Qwen2 7B, show that our method effectively enhances course-correction
skills without affecting general performance. Additionally, it effectively
improves LLMs' safety, particularly in resisting jailbreak attacks.
large language models (LLMs) course-correction C$^2$-Eval preference learning C$^2$-Syn jailbreak attacks