Open-weight AI systems offer unique benefits, including enhanced
transparency, open research, and decentralized access. However, they are
vulnerable to tampering attacks which can efficiently elicit harmful behaviors
by modifying weights or activations. Currently, there is not yet a robust
science of open-weight model risk management. Existing safety fine-tuning
methods and other post-training techniques have struggled to make LLMs
resistant to more than a few dozen steps of adversarial fine-tuning. In this
paper, we investigate whether filtering text about dual-use topics from
training data can prevent unwanted capabilities and serve as a more
tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable
data filtering and show that it offers a tractable and effective method for
minimizing biothreat proxy knowledge in LLMs. We pretrain multiple
6.9B-parameter models from scratch and find that they exhibit substantial
resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M
tokens of biothreat-related text — outperforming existing post-training
baselines by over an order of magnitude — with no observed degradation to
unrelated capabilities. However, while filtered models lack internalized
dangerous knowledge, we find that they can still leverage such information when
it is provided in context (e.g., via search tool augmentation), demonstrating a
need for a defense-in-depth approach. Overall, these findings help to establish
pretraining data curation as a promising layer of defense for open-weight AI
systems.