Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
Large Language Models (LLMs) have demonstrated great potential as generalist
assistants, showcasing powerful task understanding and problem-solving
capabilities. To deploy LLMs as AI assistants, it is crucial that these models
exhibit desirable behavioral traits, such as non-toxicity and resilience
against jailbreak attempts. Current methods for detoxification or preventing
jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement
Learning from Human Feedback (RLHF), which requires finetuning billions of
parameters through gradient descent with substantial computation cost.
Furthermore, models modified through SFT and RLHF may deviate from the
pretrained models, potentially leading to a degradation in foundational LLM
capabilities. In this paper, we observe that surprisingly, directly editing a
small subset of parameters can effectively modulate specific behaviors of LLMs,
such as detoxification and resistance to jailbreaking. Specifically, for a
behavior that we aim to avoid, we employ a linear classifier, which we term the
behavior probe, to classify binary behavior labels within the hidden state
space of the LLM. Using this probe, we introduce an algorithm to identify a
critical subset of LLM parameters that significantly influence this targeted
behavior. Then we directly edit these selected parameters by shifting them
towards the behavior probe. Such a direct parameter editing method necessitates
only inference-level computational resources. Experiments demonstrate that in
the representative detoxification task, our approach achieves reductions of up
to 90.0\% in toxicity on the RealToxicityPrompts dataset and 49.2\% on ToxiGen,
while maintaining the LLM's general capabilities in areas such as common sense,
question answering, and mathematics. Our code is available at
https://github.com/lucywang720/model-surgery.
6694bd9b64dc88cf5e83fb5a
Huanqian Wang
claimed_verified
2024-07-15T08:58:14.171Z
64ec52887e69909b1ad1ff0b
/avatars/30bb6fda51093fc71e80d0313c13d15e.svg
Lucy Wang
user
Lucywang720
6694bd9b64dc88cf5e83fb5b
Yang Yue
claimed_verified
2024-11-05T07:59:01.337Z
649d475111592b1a765ac1a3
https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
Yang Yue
user
Yang130
6694bd9b64dc88cf5e83fb5c
Rui Lu
claimed_verified
2025-04-22T10:17:38.911Z
64b77916964f6e7bf32bb428
/avatars/631b0324b4d713d351abbedb468658dc.svg
Rui Lu
user
RayLuTHU
6694bd9b64dc88cf5e83fb5d
Jingxin Shi
6694bd9b64dc88cf5e83fb5e
Andrew Zhao
claimed_verified
2024-07-16T20:16:22.404Z
630482fbce6b12280b18971d
/avatars/b07f31fd970d736bdf574d56da7a5634.svg
Andrew Zhao
user
andrewzh
6694bd9b64dc88cf5e83fb5f
Shenzhi Wang
claimed_verified
2024-07-18T09:08:06.670Z
6486dde1f74857df3f1a5828
https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
Shenzhi Wang
user
shenzhi-wang
6694bd9b64dc88cf5e83fb60
Shiji Song
6694bd9b64dc88cf5e83fb61
Gao Huang
Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
Large Language Models (LLMs) have demonstrated great potential as generalist
assistants, showcasing powerful task understanding and problem-solving
capabilities. To deploy LLMs as AI assistants, it is crucial that these models
exhibit desirable behavioral traits, such as non-toxicity and resilience
against jailbreak attempts. Current methods for detoxification or preventing
jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement
Learning from Human Feedback (RLHF), which requires finetuning billions of
parameters through gradient descent with substantial computation cost.
Furthermore, models modified through SFT and RLHF may deviate from the
pretrained models, potentially leading to a degradation in foundational LLM
capabilities. In this paper, we observe that surprisingly, directly editing a
small subset of parameters can effectively modulate specific behaviors of LLMs,
such as detoxification and resistance to jailbreaking. Specifically, for a
behavior that we aim to avoid, we employ a linear classifier, which we term the
behavior probe, to classify binary behavior labels within the hidden state
space of the LLM. Using this probe, we introduce an algorithm to identify a
critical subset of LLM parameters that significantly influence this targeted
behavior. Then we directly edit these selected parameters by shifting them
towards the behavior probe. Such a direct parameter editing method necessitates
only inference-level computational resources. Experiments demonstrate that in
the representative detoxification task, our approach achieves reductions of up
to 90.0\% in toxicity on the RealToxicityPrompts dataset and 49.2\% on ToxiGen,
while maintaining the LLM's general capabilities in areas such as common sense,
question answering, and mathematics. Our code is available at
https://github.com/lucywang720/model-surgery.
Directly editing a subset of LLM parameters can effectively modulate specific behaviors like detoxification and jailbreak resistance with minimal computational cost while maintaining general LLM capabilities.
Large Language Models LLMS Supervised Fine-Tuning SFT Reinforcement Learning from Human Feedback RLHF behavior probe hidden state space parameter editing RealToxicityPrompts dataset ToxiGen dataset