The paper introduces the Influence Approximation Unlearning (IAU) algorithm, which leverages incremental learning principles to efficiently address the computational challenges of influence-based unlearning in machine learning models.
Due to growing privacy concerns, machine unlearning, which aims at enabling
machine learning models to “forget” specific training data, has received
increasing attention. Among existing methods, influence-based unlearning has
emerged as a prominent approach due to its ability to estimate the impact of
individual training samples on model parameters without retraining. However,
this approach suffers from prohibitive computational overhead arising from the
necessity to compute the Hessian matrix and its inverse across all training
samples and parameters, rendering it impractical for large-scale models and
scenarios involving frequent data deletion requests. This highlights the
difficulty of forgetting. Inspired by cognitive science, which suggests that
memorizing is easier than forgetting, this paper establishes a theoretical link
between memorizing (incremental learning) and forgetting (unlearning). This
connection allows machine unlearning to be addressed from the perspective of
incremental learning. Unlike the time-consuming Hessian computations in
unlearning (forgetting), incremental learning (memorizing) typically relies on
more efficient gradient optimization, which supports the aforementioned
cognitive theory. Based on this connection, we introduce the Influence
Approximation Unlearning (IAU) algorithm for efficient machine unlearning from
the incremental perspective. Extensive empirical evaluations demonstrate that
IAU achieves a superior balance among removal guarantee, unlearning efficiency,
and comparable model utility, while outperforming state-of-the-art methods
across diverse datasets and model architectures. Our code is available at
https://github.com/Lolo1222/IAU.