Concept Ablation Fine-Tuning (CAFT) uses interpretability tools to steer LLM generalization away from unintended concepts without altering training data.
Fine-tuning large language models (LLMs) can lead to unintended
out-of-distribution generalization. Standard approaches to this problem rely on
modifying training data, for example by adding data that better specify the
intended generalization. However, this is not always practical. We introduce
Concept Ablation Fine-Tuning (CAFT), a technique that leverages
interpretability tools to control how LLMs generalize from fine-tuning, without
needing to modify the training data or otherwise use data from the target
distribution. Given a set of directions in an LLM’s latent space corresponding
to undesired concepts, CAFT works by ablating these concepts with linear
projections during fine-tuning, steering the model away from unintended
generalizations. We successfully apply CAFT to three fine-tuning tasks,
including emergent misalignment, a phenomenon where LLMs fine-tuned on a narrow
task generalize to give egregiously misaligned responses to general questions.
Without any changes to the fine-tuning data, CAFT reduces misaligned responses
by 10x without degrading performance on the training distribution. Overall,
CAFT represents a novel approach for steering LLM generalization without
modifying training data.