Distillation Robustifies Unlearning
Alex Infanger, Bruce W. Lee, Addie Foote, Leni Shor, Harish Kamath, Bryce Woodworth, Alex Cloud, Alexander Turner, Jacob Goldman-Wetzler
Large language models acquire undesired capabilities during pretraining, such as knowledge that could support the development of weapons of mass destruction. Machine unlearning methods aim to remove these capabilities but typically achieve only behavioral suppression: the model appears safe but retains latent capabilities that can be easily recovered through adversarial finetuning. To address this limitation, we show that distilling an unlearned model into a randomly initialized student yields a model that is significantly more robust to such relearning attacks. Next, we introduce Unlearn-Noise-Distill-on-Outputs (UNDO), which approximates network reinitialization by copying and noising the weights of an unlearned teacher model. UNDO attains most of the robustness of gold-standard data filtering at a fraction of the compute cost. In extensive comparisons to unlearning baselines on language and arithmetic tasks, UNDO achieves Pareto optimal robustness and performance. On the Weapons of Mass Destruction Proxy (WMDP) benchmark, UNDO robustifies unlearning to achieve competitive performance. Since LLM developers already apply distillation to reduce inference costs, our results show they can obtain robustness essentially “for free” by incorporating suppression before distillation. This insight offers a practical path to safer model deployment and open-weight release.