Cybersecurity and cyberattacks cost hundreds of billions of dollars annually.1 Rapid progress in AI will dramatically increase the stakes.
In the worst case, AI may greatly reduce the effort required to unleash a devastating attack on critical infrastructure systems. New AI models are already being used in the wild to increase the scope and reduce the cost of attacks,2 a trend which will only accelerate.3
Attacks on critical infrastructure are limited due to restraint on behalf of state-level actors. However, AI may soon enable unaccountable non-state actors to carry out large-scale, sophisticated attacks.
AI will also enable better cyberdefense. However, due to structural issues in the way systems are developed, deployed, and managed, we frequently fail to take advantage of existing best practices, let alone the rapidly evolving capabilities that AI will provide. To navigate the coming wave, the technology and regulatory sectors must coordinate to address these issues. In the best case, AI can serve as both an enabling technology to improve defenses, and a wake-up call to address long-standing deficiencies in our approach to security.
The Cyberattack Overhang over Critical Infrastructure
It is hardly necessary to review the scale of the cybersecurity challenge. While many attacks go unreported, there is no lack of well-known incidents, from the Chinese breach of detailed personnel records for millions of Americans working in secure positions,4 to the NotPetya attack (with damages estimated at over ten billion dollars5), to the Equifax data breach of records covering over 160 million Americans and British citizens.6 A recent attack on the US’ largest health care payment system is currently imperiling the finances of medical practices across the country.7
However, this is potentially just the tip of the iceberg. A vast swath of critical infrastructure – the electrical grid, communications systems, water treatment facilities, air traffic control, port facilities,8 military systems, and much more – relies on vulnerable systems. In 2021, a ransomware attack resulted in a major East Coast oil pipeline being shut down for several days,9 causing panic buying and long lines at gas stations in multiple states. In December 2022, Southwest Airlines’ crew scheduling system collapsed for several days,10 resulting in over 15,000 flight cancellations. Numerous government reports detail the extent of vulnerabilities in critical infrastructure systems.11 As the world becomes increasingly dependent on software systems (including increasing use of AI), a worst-case cyberattack could have severe consequences.
Disconcertingly, our lack of routine infrastructure failures seems to stem more from the reticence of potential attackers than from the inherent security of our systems. In other words, we are living under a “cyberattack overhang”.
Indeed, the US NSA and other agencies have reported on “Volt Typhoon”, an extensive Chinese effort to “preposition themselves on IT networks for disruptive or destructive cyberattacks against U.S. critical infrastructure”.12 Targets included ports, energy and water controls,13 often near military bases. Increasingly capable AI will expand the potential scope of such attacks, as well as empowering non-state actors who might be less deterred by the consequences of an infrastructure attack. Given the difficulty of identifying the ultimate perpetrator of a cyberattack, the results could be destabilizing.
AI Impacts on Cyber Offense and Defense
AI has many foreseeable applications to cybersecurity. Some capabilities will primarily help attackers, while others will help defenders.
On the offensive side, AI will soon plausibly be able to automate the entire attack chain,14 from intelligence gathering (analyzing public information to identify the software used by a target), to invoking a known exploit against the target software, and analyzing data in the target system to determine next steps.15 Google projects that within the year, AI will also be used to scale “social engineering” attacks16 (tricking personnel into revealing information or otherwise assisting an attacker); given the facility of large language models at crafting persuasive prose, this could have a large impact.
Conversely, AI should be able to automate many aspects of defense, such as detecting and correcting unpatched or misconfigured software. AI techniques are already used for “anomaly detection” – identifying unusual behavior that might indicate an attack.17 Coding assistants may eventually be able to automatically detect more bugs, as well as help “harden” software by rewriting it to use more secure languages, libraries, and techniques, potentially including mathematical proofs of correctness.
In principle, progress might, on balance, favor defense. A system designed and operated by an ideal defender would have no vulnerabilities, leaving even an ideal attacker unable to break in.18 Also, AI works best when given large amounts of data to work with, and defenders generally have access to more data.19 However, absent substantial changes to cyber practices, we are likely to see many dramatic AI-enabled incidents.
The primary concern is that advances in defensive techniques are of no help if defenders are not keeping up to date. Despite decades of effort, it is well known that important systems are often misconfigured and/or running out-of-date software.20 For instance, a sensitive application operated by credit report provider Equifax was found in 2017 to be accessible to anyone on the Internet, simply by typing “admin” into the login and password fields.21 A recent CISA report notes that this government agency often needs to resort to subpoenas merely to identify the owners of vulnerable infrastructure systems, and that most issues they detect are not remediated in the same year.
As AI enables increasingly sophisticated, large-scale, fast-moving attacks, defenders will need to move faster than ever to keep up. However, millions of individuals are in some fashion responsible for the security of a digital system. Experience shows that we cannot rely on everyone to universally follow best practices, especially given the many practical difficulties that such practices often entail.22 To prevent AI from enabling a tidal wave of cyberattacks, and in particular to have any practical hope of securing our many critical infrastructure systems against increasingly capable attackers, we must find ways to shift the playing field.
Mitigation
As cyberattacks become even more prevalent and sophisticated, it will be necessary to apply defenses in a more systematic fashion. In this section, we briefly present some potential approaches, with an emphasis on approaches that can leverage AI, asymmetrically benefit defenders, and reduce the burden on individual system operators.
Stronger Foundations
The best remedy for a security flaw is to prevent the flaw from existing in the first place. Modern coding practices23 can help reduce the number of exploitable software bugs, but these practices may involve additional effort and/or require rewriting older software. AI-based tools can assist with this work.24
Many security breaches owe as much to user error as to software bugs. New technologies such as passkeys eliminate the possibility of users employing weak passwords or being tricked (“phished”) into revealing their password to an attacker.
Systematic Defense
Rather than relying entirely on overburdened system operators to individually maintain the highest standards of security, we should look for opportunities to supplement security with systematic approaches.
Attackers perform “vulnerability scans” to locate misconfigured or out-of-date servers, or sensitive information that has accidentally been placed in public view. If an attacker can find a vulnerability, a defender should be able to find it first.25 In particular, we should enable “good guys” to systematically scan the Internet to identify and remediate vulnerabilities. Realizing this in practice will require addressing a number of practical, organizational, and legal challenges,26 but companies such as Google are already performing similar activities as a public service.27 In a related practice, “Dark web monitoring” can systematically monitor data leaks if the data appears on the dark web.28
Because it is difficult to maintain systems in a state of perfect security, operators often use firewalls (which limit access to potentially vulnerable systems), signature scanning (watching for known malicious software,29 as well as known vulnerabilities) and anomaly detection software (which looks for unusual access patterns that might indicate a security breach).30 Cloud computing platforms, software-as-a-service providers, and networking equipment could provide more such functionality. This would facilitate additional, constantly-updated, professionally managed security by default.31 The introduction of firewalls into PCs and Internet service providers (ISPs) is one of the main reasons that Internet worms like Conficker are no longer prevalent.32
Facilitating Security Patches
In a world where the ability of attackers to identify and exploit ever-more-subtle bugs is continually advancing, it is critical that security patches be applied in a timely manner.33 Unfortunately, as mentioned earlier, this is not always possible. Even if the bug fix itself was small, an entirely new version of the software must be created, tested, and installed – a significant burden for both the software provider and the user, and possibly subject to regulatory hurdles.
We must seek out new approaches to software design and distribution that facilitate the application of security patches,34 and regulations must be updated to streamline such updates. Solutions must encompass scenarios where a system manufacturer (or one of their suppliers) has gone out of business.
Safety Culture
Cybersecurity suffers from a lack of transparency. It is difficult to tell which organizations follow good practices; security lapses are often not reported, unless they lead to an outright breach which impacts customers. This reduces the impetus to prioritize cybersecurity, especially because the impact of a breach often falls heavily on third parties. (For instance, when a company fails to secure customer information, it is the customers who are vulnerable to identity theft.)
Contrast this with the airline industry, where a strict safety culture, including stringent reporting requirements and blameless investigation practices, covering near-misses and procedural failures as well as full-blown accidents, have yielded a remarkable level of safety despite the inherent complexity of air travel.
Cybersecurity can be enhanced through strict requirements for reporting security lapses, along with whistleblower provisions to ensure compliance. As in the airline industry, the focus should be on learning from breaches and near-misses, rather than assigning blame.
Other forms of transparency can create social pressure to adhere to good security practices, including the other measures described here. For instance, rating agencies could be established to evaluate software providers on timeliness of security updates, and cloud operators according to the quality of their internal security and degree of assistance with customer security. Whenever possible, ratings should be based on outcome metrics, rather than adherence to specific practices that might not always correlate with actual security.
Responsible Release of Dangerous Capabilities
No matter how much we manage to strengthen defenses, we should still attempt to minimize the potential for increasingly advanced AI to assist attackers.
Many capabilities are “dual-use”, i.e. of value to both attackers and defenders. The release of new tools that advance such capabilities should follow responsible disclosure policies, allowing time for vulnerabilities to be patched before attackers gain access. Research into automatic patching of vulnerabilities is also called for.
General-purpose AIs should be designed to refuse requests to assist in cyberattacks.35 However, it is difficult to impose such restrictions in a robust manner, especially for open-source models, which is why responsible disclosure is always important.
Advanced models should be rigorously evaluated to determine whether they provide new capabilities for attackers. Release of such a model should be delayed until defensive measures can be updated (for example, release of advances in bug detection should be delayed until they can be applied in private to widely used software packages, and any issues found have been fixed and the patches deployed widely).
Conclusion
Virtually every aspect of modern life, from the operations of corporations large and small to critical energy, transportation, water, and other infrastructure, rely on systems that are vulnerable to cyberattack. By default, progress will leave us open to devastating attacks, as attackers will quickly make use of new AI capabilities but defenders often lag behind. Our current sense of relative security depends in part on the reluctance of state-level actors to cause visible damage, but as AI provides leverage to smaller actors, this “cyberattack overhang” could translate into startling consequences.
To mitigate this danger, we must regulate development of, and access to, AIs with dangerous capabilities. However, this cannot be our only line of defense. Legacy software must migrate to modern, safe coding practices; we must shift responsibility for security from individual system operators to professional organizations; we must move away from the assumption that all systems can be frequently updated with security patches. This will require coordinated efforts across the entire technology sector. The advent of advanced AI must be our wake-up call to finally address long-standing issues in our approach to cybersecurity.
Guest author Steve Newman, a co-founder of eight startups including Google Docs (née Writely), is now searching for ways to reduce the tension between progress and safety by building a more robust world. His blog is Am I Stronger Yet?
Thanks to Brendan Dolan-Gavitt, Dan Hendrycks, Fish Wang, Ido Yariv, Mark Bailey, Massimiliano Poletto, Michael Chen, Nathaniel Li, Will Hodgkins, and Yan Shoshitaishvili for contributions and feedback. No endorsement is implied.
