CRISP: Persistent Concept Unlearning via Sparse Autoencoders
As large language models (LLMs) are increasingly deployed in real-world
applications, the need to selectively remove unwanted knowledge while
preserving model utility has become paramount. Recent work has explored sparse
autoencoders (SAEs) to perform precise interventions on monosemantic features.
However, most SAE-based methods operate at inference time, which does not
create persistent changes in the model's parameters. Such interventions can be
bypassed or reversed by malicious actors with parameter access. We introduce
CRISP, a parameter-efficient method for persistent concept unlearning using
SAEs. CRISP automatically identifies salient SAE features across multiple
layers and suppresses their activations. We experiment with two LLMs and show
that our method outperforms prior approaches on safety-critical unlearning
tasks from the WMDP benchmark, successfully removing harmful knowledge while
preserving general and in-domain capabilities. Feature-level analysis reveals
that CRISP achieves semantically coherent separation between target and benign
concepts, allowing precise suppression of the target features.
68a6fc7d9e4b49496aac6b8c
Tomer Ashuach
claimed_verified
2025-08-25T08:27:35.955Z
638dbf006b5c2ccc6240d6fc
https://cdn-avatars.huggingface.co/v1/production/uploads/638dbf006b5c2ccc6240d6fc/7jZlE4S35f15PLSN3_bVD.jpeg
Tomer Ashuach
user
Tomertech
68a6fc7d9e4b49496aac6b8d
Dana Arad
68a6fc7d9e4b49496aac6b8e
Aaron Mueller
68a6fc7d9e4b49496aac6b8f
Martin Tutek
68a6fc7d9e4b49496aac6b90
Yonatan Belinkov
CRISP: Persistent Concept Unlearning via Sparse Autoencoders
As large language models (LLMs) are increasingly deployed in real-world
applications, the need to selectively remove unwanted knowledge while
preserving model utility has become paramount. Recent work has explored sparse
autoencoders (SAEs) to perform precise interventions on monosemantic features.
However, most SAE-based methods operate at inference time, which does not
create persistent changes in the model's parameters. Such interventions can be
bypassed or reversed by malicious actors with parameter access. We introduce
CRISP, a parameter-efficient method for persistent concept unlearning using
SAEs. CRISP automatically identifies salient SAE features across multiple
layers and suppresses their activations. We experiment with two LLMs and show
that our method outperforms prior approaches on safety-critical unlearning
tasks from the WMDP benchmark, successfully removing harmful knowledge while
preserving general and in-domain capabilities. Feature-level analysis reveals
that CRISP achieves semantically coherent separation between target and benign
concepts, allowing precise suppression of the target features.
CRISP is a parameter-efficient method using sparse autoencoders to permanently remove unwanted knowledge from large language models while preserving their utility.
sparse autoencoders parameter-efficient concept unlearning salient features feature-level analysis WMDP benchmark semantically coherent separation