Scrub It Out! Erasing Sensitive Memorization in Code Language Models via
  Machine Unlearning
While Code Language Models (CLMs) have demonstrated superior performance in
software engineering tasks such as code generation and summarization, recent
empirical studies reveal a critical privacy vulnerability: these models exhibit
unintended memorization of sensitive training data, enabling verbatim
reproduction of confidential information when specifically prompted. To address
this issue, several approaches, including training data de-duplication and
differential privacy augmentation, have been proposed. However, these methods
require full-model retraining for deployed CLMs, which incurs substantial
computational costs. In this paper, we aim to answer the following research
question: Can sensitive information memorized by CLMs be erased effectively and
efficiently?
  We conduct a pioneering investigation into erasing sensitive memorization in
CLMs through machine unlearning - a post-hoc modification method that removes
specific information from trained models without requiring full retraining.
Specifically, we first quantify the memorization risks of sensitive data within
CLM training datasets and curate a high-risk dataset of 50,000 sensitive
memorized samples as unlearning targets. We study two widely used gradient
ascent-based unlearning approaches: the vanilla and constraint-based methods,
and introduce CodeEraser, an advanced variant that selectively unlearns
sensitive memorized segments in code while preserving the structural integrity
and functional correctness of the surrounding code. Extensive experiments on
three families of CLMs, i.e., CodeParrot, CodeGen-Mono, and Qwen2.5-Coder,
validate the effectiveness and efficiency of CodeEraser in erasing targeted
sensitive memorization while maintaining model utility.
68cb88205a7803ff3be42e43
Zhaoyang Chu
68cb88205a7803ff3be42e44
Yao Wan
68cb88205a7803ff3be42e45
Zhikun Zhang
68cb88205a7803ff3be42e46
Di Wang
68cb88205a7803ff3be42e47
Zhou Yang
68cb88205a7803ff3be42e48
Hongyu Zhang
68cb88205a7803ff3be42e49
Pan Zhou
68cb88205a7803ff3be42e4a
Xuanhua Shi
68cb88205a7803ff3be42e4b
Hai Jin
68cb88205a7803ff3be42e4c
David Lo
Scrub It Out! Erasing Sensitive Memorization in Code Language Models via
  Machine Unlearning
While Code Language Models (CLMs) have demonstrated superior performance in
software engineering tasks such as code generation and summarization, recent
empirical studies reveal a critical privacy vulnerability: these models exhibit
unintended memorization of sensitive training data, enabling verbatim
reproduction of confidential information when specifically prompted. To address
this issue, several approaches, including training data de-duplication and
differential privacy augmentation, have been proposed. However, these methods
require full-model retraining for deployed CLMs, which incurs substantial
computational costs. In this paper, we aim to answer the following research
question: Can sensitive information memorized by CLMs be erased effectively and
efficiently?
  We conduct a pioneering investigation into erasing sensitive memorization in
CLMs through machine unlearning - a post-hoc modification method that removes
specific information from trained models without requiring full retraining.
Specifically, we first quantify the memorization risks of sensitive data within
CLM training datasets and curate a high-risk dataset of 50,000 sensitive
memorized samples as unlearning targets. We study two widely used gradient
ascent-based unlearning approaches: the vanilla and constraint-based methods,
and introduce CodeEraser, an advanced variant that selectively unlearns
sensitive memorized segments in code while preserving the structural integrity
and functional correctness of the surrounding code. Extensive experiments on
three families of CLMs, i.e., CodeParrot, CodeGen-Mono, and Qwen2.5-Coder,
validate the effectiveness and efficiency of CodeEraser in erasing targeted
sensitive memorization while maintaining model utility.
CodeEraser effectively and efficiently removes sensitive memorized information from Code Language Models using machine unlearning techniques without full retraining.
Code Language Models CLMs machine unlearning gradient ascent CodeParrot CodeGen-Mono Qwen2.5-Coder CodeEraser