SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder
  Guardrails for Precision Unlearning in LLMs
Machine unlearning is a promising approach to improve LLM safety by removing
unwanted knowledge from the model. However, prevailing gradient-based
unlearning methods suffer from issues such as high computational costs,
hyperparameter instability, poor sequential unlearning capability,
vulnerability to relearning attacks, low data efficiency, and lack of
interpretability. While Sparse Autoencoders are well-suited to improve these
aspects by enabling targeted activation-based unlearning, prior approaches
underperform gradient-based methods. This work demonstrates that, contrary to
these earlier findings, SAEs can significantly improve unlearning when employed
dynamically. We introduce Dynamic DAE Guardrails (DSG), a novel
method for precision unlearning that leverages principled feature selection and
a dynamic classifier. Our experiments show DSG substantially outperforms
leading unlearning methods, achieving superior forget-utility trade-offs. DSG
addresses key drawbacks of gradient-based approaches for unlearning -- offering
enhanced computational efficiency and stability, robust performance in
sequential unlearning, stronger resistance to relearning attacks, better data
efficiency including zero-shot settings, and more interpretable unlearning.
67fcb3584a92187863e732d5
Aashiq Muhamed
claimed_verified
2025-05-27T07:55:50.625Z
64755a83e0b188d3cb2579d8
/avatars/2c50590905f4bd398a4c9991e1b4b5bb.svg
Aashiq Muhamed
user
aashiqmuhamed
67fcb3584a92187863e732d6
Jacopo Bonato
67fcb3584a92187863e732d7
Mona Diab
67fcb3584a92187863e732d8
Virginia Smith
SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder
  Guardrails for Precision Unlearning in LLMs
Machine unlearning is a promising approach to improve LLM safety by removing
unwanted knowledge from the model. However, prevailing gradient-based
unlearning methods suffer from issues such as high computational costs,
hyperparameter instability, poor sequential unlearning capability,
vulnerability to relearning attacks, low data efficiency, and lack of
interpretability. While Sparse Autoencoders are well-suited to improve these
aspects by enabling targeted activation-based unlearning, prior approaches
underperform gradient-based methods. This work demonstrates that, contrary to
these earlier findings, SAEs can significantly improve unlearning when employed
dynamically. We introduce Dynamic DAE Guardrails (DSG), a novel
method for precision unlearning that leverages principled feature selection and
a dynamic classifier. Our experiments show DSG substantially outperforms
leading unlearning methods, achieving superior forget-utility trade-offs. DSG
addresses key drawbacks of gradient-based approaches for unlearning -- offering
enhanced computational efficiency and stability, robust performance in
sequential unlearning, stronger resistance to relearning attacks, better data
efficiency including zero-shot settings, and more interpretable unlearning.
Dynamic DAE Guardrails improve machine unlearning in large language models by overcoming limitations of gradient-based methods and offering enhanced efficiency, stability, and interpretability.
Sparse Autoencoders Dynamic DAE Guardrails precision unlearning feature selection dynamic classifier forget-utility trade-offs sequential unlearning relearning attacks data efficiency zero-shot settings interpretability