Recent decades have seen massive amounts of biological and medical data becoming available in digital form. The computerization of lab equipment, digitization of medical records, and advent of cheap DNA sequencing all generate data, which is increasingly collected in large data sets available to researchers.
This bounty of data is driving rapid progress in AI. In combination with increasingly cheap and powerful DNA synthesis and laboratory automation, AI promises to deliver revolutionary advances in medicine via rapid design-build-test cycles. However, the same capabilities that are driving everything from drug discovery to diagnostic techniques could also revolutionize bioterrorism, with plausible scenarios ranging from a flu-level virus to a pandemic with an impact exceeding Covid.
Fortunately, there is much that can be done to mitigate the risk of an AI-assisted pandemic. This is often framed as a conflict between progress and safety, but that needn’t be the case. Many mitigations are simply sensible public health measures, such as improved ventilation in public spaces. Where it is necessary to manage the development of AI capabilities, this can be done in a targeted manner. We have the opportunity to simultaneously improve public health, reduce the risk of pandemics, and smooth the path for progress.
How AI Could Enable Bioterrorism
There are multiple paths through which advances in AI could lead to the deliberate release of a harmful virus. One scenario hinges on the fact, not widely recognized, that the genetic sequences for tens of thousands of human viruses (presenting varying degrees of danger) are already widely available.1 Advances in DNA/RNA synthesis make it theoretically possible for a disturbed or radicalized individual to recreate a virus; the greatest barrier is knowledge of lab techniques.2 For many viruses, that barrier is surprisingly low. If the individual had managed to identify a particularly dangerous virus, tens or even hundreds of thousands of deaths could result, comparable to the atomic bombs dropped on Hiroshima and Nagasaki.3
Future chatbots seem likely to be capable of lowering the bar to such an attack. As models become increasingly “multimodal”, their training data will soon include video, such as university lectures and lab demonstrations. Such systems would not be limited to providing written instructions; they could plausibly use a camera to observe a would-be terrorist’s work and coach them through each step of viral synthesis. Future models (if not mitigated) also seem likely to be able to provide meaningful help in planning attacks, brainstorming everything from general planning, to obtaining equipment, to applying published research toward creating more-hazardous viruses, to where and how to release a virus to cause maximum impact.
It is sometimes suggested that these systems won’t make a meaningful difference, because the information they are trained on is already public. However, the runaway success of chatbots stems from their ability to surface the right information at the right time. Google can provide a list of ten web pages that are loosely related to a desired topic; ChatGPT can interpolate between vast amounts of training data to provide precisely the information a user needs, even if they don’t properly know how to ask for it. For instance, a multimodal AI might be able to point out to a would-be bioterrorist that they are committing an error in lab technique that would contaminate their sample and ruin their attempt at viral synthesis.
Developments in biology could further raise the stakes. Ongoing gain-of-function research could yield the genome for an unusually dangerous virus; or field work could uncover a dangerous animal virus. Alternatively, progressing in characterizing viruses’ pandemic potential might eventually allow an attacker to select the most dangerous of the existing public genomes for synthesis.
Finally, we must consider the possibility that future specialized biotechnology AIs will be able to support the design of proteins having specific desired behaviors in the body.4 This possibility underlies much of the hoped-for potential of AI to revolutionize medicine, but if protein design tools eventually progress as far as some proponents envision, they could be used to increase the transmissibility or lethality of a virus. This could abet the creation of a “supervirus” – say, combining the rapid spread of measles, the fatality rate of smallpox, and/or the incubation period of HIV.5 The creator of such a virus could then release it under circumstances that allow it to disperse widely before first being detected. It is plausible that the impact would be so severe that people might be afraid to leave their homes, and essential workers might be unable or unwilling to remain on the job, conceivably leading to a breakdown of civilization.
General Mitigations for Respiratory Viruses
The danger rests on the fact that modern society is highly vulnerable to respiratory viruses in general. Endemic viruses such as flu, RSV, and SARS-CoV-26 cause over half a million deaths per year. Actions which make it more difficult for viruses to propagate will yield health and economic benefits today, in addition to reducing the risk of bioterrorism.
Improvements to air ventilation, filtration, and germicidal ultraviolet lighting make it more difficult for respiratory viruses to travel from one person to another. UV lighting in particular is an area of active research with considerable potential to reduce viral circulation.7
Development of broad-spectrum vaccines and antivirals8 could reduce the impact of common viral families such as flu and coronaviruses.9 This would reduce the potential for a bad actor to leverage the extensive genetic sequences and knowledge base around these families.
General Mitigations for Pandemics
Early-detection and rapid-response capabilities can reduce the impact of both engineered and natural pandemics. The Covid pandemic killed over six million people,10 and the economic impact is measured in trillions of dollars; some actions to reduce the potential for another pandemic could be carried out at comparatively low cost. The next Covid might be a century away, or Patient Zero might already be developing symptoms today.
Aggressive monitoring for novel viruses could detect new viruses before they are able to spread widely. Wastewater monitoring (especially targeting airports and other travel hubs)11 may be particularly effective, but should be combined with other measures, as not all viruses present in wastewater.
Build the capability to very quickly manufacture and distribute test kits once a new virus has been identified. Aggressive testing around early cases can help prevent a virus from establishing itself.
Develop improved PPE12 for airborne viruses, targeting cost/durability, effectiveness, and practicality/comfort. Stockpile PPE for rapid deployment to essential workers.
Further accelerate our ability to rapidly create, test, manufacture, and distribute a vaccine for a novel virus.13
Preventing a Deliberate Virus Release
As AIs become more powerful, we will need to carefully manage access to tools that could assist in causing harm, as well as information regarding dangerous viral genomes.14 Restrictions can be designed to minimize impact on legitimate research, but in some cases there will be tradeoffs to be made.
Monitor patterns of behavior in users of biotech tools, to identify individuals who may be attempting to create a dangerous virus.15 Encourage reporting of suspicious behavior.16
Limit access to tools needed for creating, synthesizing, or testing a virus,17 such as specialized AI models (e.g. protein design tools), DNA / RNA synthesis equipment, and other specialized equipment. Measures should include “know your customer” requirements, tracking of equipment over time, and comprehensive screening of synthesized DNA / RNA sequences. If and when protein design tools become capable of enabling the creation of novel viruses, screening will need to be expanded to detect such novel viruses (a potentially difficult problem).18
Develop techniques for detecting “warning shots”. A failed attempt at engineering a pandemic might sicken a small number of people (perhaps the perpetrator). Techniques for identifying novel, suspicious viruses could allow us to head off a successful attack.
Exclude certain categories of biological knowledge19 from chatbots and other widely accessible AIs, so as to prevent them from coaching a malicious actor through the creation of a virus. Access to AIs with hazardous knowledge should be restricted to vetted researchers.20
Evaluations and red-teaming to prevent the release of AI models that can assist with the synthesis and release of a virus.21 Training an AI to assist with the development of new viruses will likely require assembling large amounts of data regarding the behavior of viruses in the body, so the development of such data sets should be monitored.
Limit (to the legitimate research community) access to genetic sequences or other specific information which would identify a specific pathogen as potentially capable of generating a pandemic, and facilitating the synthesis of that pathogen.22
Limit research into techniques for evaluating the potential for viruses to cause harm in humans, particularly with regard to transmissibility. Especially limit the open publication of such research.
Apply rigorous risk-benefit analysis to viral gain-of-function research, including the decision of whether to openly publish the results. This analysis should take into account anticipated developments in synthesis techniques. For instance, if a virus might plausibly be easy to synthesize using the equipment that will plausibly be available ten years from now, then unrestricted publication of the viral genome today might be considered high risk.
These measures will require identifying legitimate researchers, and restricting access to certain narrow categories of information and tools to people in that circle. Maintaining such restrictions in an effective manner will require new practices in certain scientific and engineering communities.
Fostering Security Mindset For Biological Research
Restricted access is at odds with the established practices and norms in most scientific fields. Traditionally, the modern academic enterprise is built around openness; the purpose of academic research is to publish, and thus contribute to our collective understanding. Sometimes norms need to adapt to changing circumstances,23 but this is never easy.
It is worth noting that virology, like many fields, is already under a heavy regulatory burden, designed to protect both research subjects and the eventual consumers of new drugs and procedures. Adding to this burden should not be taken lightly, but in some cases will be necessary. Historically, regulations and norms have not always been designed to provide the maximum protection in return for the minimum impact on research. Meanwhile, the stakes are higher than ever before: the impact of a new pandemic could be vastly greater than that of an improperly approved drug. And if we wait until the first cases of an engineered virus become visible before applying restrictions on research, that will be much too late to head off the resulting pandemic.
The important question is whether or not we succeed in preventing an engineered pandemic, as opposed to merely adhering to regulations. Effective biosecurity will require helping the scientific community to adopt a security mindset, educating them on the principles of security and making them into enthusiastic and active participants.24 Measures might include:
Developing specific regulations and best practices, updated on a regular basis. Transparent measurement of security effectiveness in practice, including “red teaming”. For instance, measure the effectiveness of DNA synthesis services at rejecting dangerous sequences,25 the difficulty of evading know-your-customer measures, and the ability of AIs to assist with lab procedures required to synthesize a virus.Promoting a risk-benefit approach to evaluating research projects.Incorporating security and risk management practices26 into undergraduate education, funding criteria, and publication criteria.Education regarding the responsibilities of the scientific community and how to fulfill them, so that bioscientists and AI developers can not only follow the letter of these procedures and guidelines, but support their spirit, guarding against developments or loopholes that could allow a malicious actor to bypass security mechanisms.Establish a support center to provide advice for how to maintain security in specific situations that arise in practice.27
Conclusion
Advances in AI, DNA synthesis, and laboratory automation promise to revolutionize medicine… but could also open the door to bioterrorism. Through a thoughtful mix of public health measures and targeted management of access to advanced capabilities, we can not only manage this risk, but also reduce the ongoing burden of natural disease.
Guest author Steve Newman, a co-founder of eight startups including Google Docs (née Writely), is now searching for ways to reduce the tension between progress and safety by building a more robust world. His blog is Am I Stronger Yet?
Thanks to Aidan O’Gara, Dan Hendrycks, Geetha Jeyapragasan, Gigi Gronvall, Lennart Justen, Mantas Mazeika, Nikki Teran, Rahul Arora, Rocco Casagrande, Sarah Carter, and Thomas Woodside for contributions and feedback. No endorsement is implied.
