There is a newly identified risk that creators of open-source LLMs can extract fine-tuning data from downstream models through backdoor training, even with black-box access.
Fine-tuning on open-source Large Language Models (LLMs) with proprietary data
is now a standard practice for downstream developers to obtain task-specific
LLMs. Surprisingly, we reveal a new and concerning risk along with the
practice: the creator of the open-source LLMs can later extract the private
downstream fine-tuning data through simple backdoor training, only requiring
black-box access to the fine-tuned downstream model. Our comprehensive
experiments, across 4 popularly used open-source models with 3B to 32B
parameters and 2 downstream datasets, suggest that the extraction performance
can be strikingly high: in practical settings, as much as 76.3% downstream
fine-tuning data (queries) out of a total 5,000 samples can be perfectly
extracted, and the success rate can increase to 94.9% in more ideal settings.
We also explore a detection-based defense strategy but find it can be bypassed
with improved attack. Overall, we highlight the emergency of this newly
identified data breaching risk in fine-tuning, and we hope that more follow-up
research could push the progress of addressing this concerning risk. The code
and data used in our experiments are released at
https://github.com/thu-coai/Backdoor-Data-Extraction.