Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging
Fine-tuning large language models (LLMs) for downstream tasks is a widely
adopted approach, but it often leads to safety degradation in safety-aligned
LLMs. Currently, many solutions address this issue by incorporating additional
safety data, which can be impractical in many cases. In this paper, we address
the question: How can we improve downstream task performance while preserving
safety in LLMs without relying on additional safety data? We propose a simple
and effective method that maintains the inherent safety of LLMs while enhancing
their downstream task performance: merging the weights of pre- and
post-fine-tuned safety-aligned models. Experimental results across various
downstream tasks, models, and merging methods demonstrate that this approach
effectively mitigates safety degradation while improving downstream task
performance, offering a practical solution for adapting safety-aligned LLMs.
67727ca4986fbffa7a2208d4
Hua Farn
extracted_pending
2024-12-30T10:57:42.126Z
641a9cb1f9dd6391a2463477
/avatars/73caf5389f88c7356f02ee9f73146faa.svg
Farn Hua
user
farnhua
67727ca4986fbffa7a2208d5
Hsuan Su
claimed_verified
2024-12-30T19:29:21.989Z
608abf1272b50b02c4b02865
https://cdn-avatars.huggingface.co/v1/production/uploads/1619708309549-608abf1272b50b02c4b02865.jpeg
Hsuan Su
user
jacksukk
67727ca4986fbffa7a2208d6
Shachi H Kumar
67727ca4986fbffa7a2208d7
Saurav Sahay
admin_assigned
2024-12-30T20:52:51.579Z
626c5d06f451470f86252659
/avatars/41078e5f99a7660b979c8cf7a63d71c6.svg
Saurav Sahay
user
sauravsahay
67727ca4986fbffa7a2208d8
Shang-Tse Chen
67727ca4986fbffa7a2208d9
Hung-yi Lee
admin_assigned
2024-12-30T20:53:02.019Z
629b6eb3ad498ab19282ae6f
/avatars/adade8c512122120b06e5c50724a178d.svg
Hung-yi Lee
user
hungyilee
Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging
Fine-tuning large language models (LLMs) for downstream tasks is a widely
adopted approach, but it often leads to safety degradation in safety-aligned
LLMs. Currently, many solutions address this issue by incorporating additional
safety data, which can be impractical in many cases. In this paper, we address
the question: How can we improve downstream task performance while preserving
safety in LLMs without relying on additional safety data? We propose a simple
and effective method that maintains the inherent safety of LLMs while enhancing
their downstream task performance: merging the weights of pre- and
post-fine-tuned safety-aligned models. Experimental results across various
downstream tasks, models, and merging methods demonstrate that this approach
effectively mitigates safety degradation while improving downstream task
performance, offering a practical solution for adapting safety-aligned LLMs.
Merging weights of pre- and post-fine-tuned safety-aligned language models improves downstream task performance while preserving safety without additional safety data.
large language models fine-tuning safety degradation safety-aligned models merging weights downstream tasks