BiasEdit: Debiasing Stereotyped Language Models via Model Editing
Previous studies have established that language models manifest stereotyped
biases. Existing debiasing strategies, such as retraining a model with
counterfactual data, representation projection, and prompting often fail to
efficiently eliminate bias or directly alter the models' biased internal
representations. To address these issues, we propose BiasEdit, an efficient
model editing method to remove stereotypical bias from language models through
lightweight networks that act as editors to generate parameter updates.
BiasEdit employs a debiasing loss guiding editor networks to conduct local
edits on partial parameters of a language model for debiasing while preserving
the language modeling abilities during editing through a retention loss.
Experiments on StereoSet and Crows-Pairs demonstrate the effectiveness,
efficiency, and robustness of BiasEdit in eliminating bias compared to
tangental debiasing baselines and little to no impact on the language models'
general capabilities. In addition, we conduct bias tracing to probe bias in
various modules and explore bias editing impacts on different components of
language models.
67d125362da8f91f8ef0412f
Xin Xu
claimed_verified
2025-03-12T08:36:07.130Z
6190ab805ca89a28e9f66873
/avatars/3c7ecc398fbf851acd2a132e947a92be.svg
Xin Xu
user
XinXuNLPer
67d125362da8f91f8ef04130
Wei Xu
67d125362da8f91f8ef04131
Ningyu Zhang
claimed_verified
2025-05-21T12:27:24.087Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
67d125362da8f91f8ef04132
Julian McAuley
BiasEdit: Debiasing Stereotyped Language Models via Model Editing
Previous studies have established that language models manifest stereotyped
biases. Existing debiasing strategies, such as retraining a model with
counterfactual data, representation projection, and prompting often fail to
efficiently eliminate bias or directly alter the models' biased internal
representations. To address these issues, we propose BiasEdit, an efficient
model editing method to remove stereotypical bias from language models through
lightweight networks that act as editors to generate parameter updates.
BiasEdit employs a debiasing loss guiding editor networks to conduct local
edits on partial parameters of a language model for debiasing while preserving
the language modeling abilities during editing through a retention loss.
Experiments on StereoSet and Crows-Pairs demonstrate the effectiveness,
efficiency, and robustness of BiasEdit in eliminating bias compared to
tangental debiasing baselines and little to no impact on the language models'
general capabilities. In addition, we conduct bias tracing to probe bias in
various modules and explore bias editing impacts on different components of
language models.
BiasEdit efficiently removes stereotypical bias from language models using lightweight editor networks, preserving language modeling abilities and demonstrating superior effectiveness compared to existing methods.
language models stereotyped biases retraining counterfactual data representation projection prompting BiasEdit editor networks parameter updates debiasing loss retention loss StereoSet Crows-Pairs bias tracing