As large language models (LLMs) are increasingly deployed in real-world
applications, the need to selectively remove unwanted knowledge while
preserving model utility has become paramount. Recent work has explored sparse
autoencoders (SAEs) to perform precise interventions on monosemantic features.
However, most SAE-based methods operate at inference time, which does not
create persistent changes in the model’s parameters. Such interventions can be
bypassed or reversed by malicious actors with parameter access. We introduce
CRISP, a parameter-efficient method for persistent concept unlearning using
SAEs. CRISP automatically identifies salient SAE features across multiple
layers and suppresses their activations. We experiment with two LLMs and show
that our method outperforms prior approaches on safety-critical unlearning
tasks from the WMDP benchmark, successfully removing harmful knowledge while
preserving general and in-domain capabilities. Feature-level analysis reveals
that CRISP achieves semantically coherent separation between target and benign
concepts, allowing precise suppression of the target features.