Executive Summary: AI-lethal autonomous weapon systems are likely to begin seeing the battlefield or widespread adoption in the near future. However, AWS possesses the major risk of an AWS failure causing an accidental escalation or conflict, due to their novelty, speed, and ambiguity. I recommend that the U.S. begins publicizing instances of AWS failures and their locations as a confidence-building measure norm. While there are some drawbacks to publishing AWS failures and locations, the U.S. has previously demonstrated with nuclear arms treaties that it is willing to make these tradeoffs for the sake of stability and security.
Background:
Over the past few years, there have been large strides in AI and AI-related technologies. As these advances in AI show no sign of slowing, there has been much speculation, theories, and experimentation by people on their possible uses. One of the areas that AI is being speculated and experimented on is in combat or military uses, exemplified by the ongoing Russo-Ukrainian war.[1] AI-powered lethal autonomous weapon system has been one such area of speculation and experimentation of AI. Militaries and countries around the world including the U.S., Russia, and China have been experimenting with AI-powered lethal autonomous weapon systems[2]. The U.S. Department of Defense has already published a document relating to the integration and standards of AI as well as autonomous weapon systems in combat or the military[3]. Weapon systems like the CIWS/Phalanx systems are already able to operate semi-autonomously while Israel’s Harpy Airborne Anti-Radiation Drone is already capable of full autonomy[4]. It is only a matter of time until the widespread proliferation of lethal autonomous weapon systems
What Are AI-Powered Lethal Autonomous Weapon Systems
Lethal autonomous weapon systems (LAWS) are defined by the DOD as “A weapon system that, once activated, can select and engage targets without further intervention by an operator”.[5] The DoD has also described multiple tiers of autonomy for lethal autonomous weapon systems.[6] The lower tier of semi-autonomous weapon systems is defined by the DoD as “Semi-autonomous weapon systems, which require human operator selection and authorization to engage specific targets (e.g., human-in-the-loop control)”.[7]
A human-supervised lethal autonomous weapon system is defined by the DoD as “Human-supervised autonomous weapon systems, which allow human intervention and, if needed, termination of the engagement, with the exception of time-critical attacks on platforms or installations (e.g., human on-the-loop control)”.[8] A fully autonomous lethal weapon system is a weapon system that can and will operate without any human intervention or supervision. All lethal autonomous weapon systems are powered or controlled by artificial intelligence.
How Can AWS Fail
There are various risks of AWS, one of which is the chance of accidental escalations or conflict. AWS like human beings are not infallible and are capable of failures, glitches, malfunctions, and mistakes. These failures, glitches, malfunctions, or mistakes can take many different forms such as engaging with supposed targets when they were not supposed to, crossing a border during a tense period, killing civilians, visibly targeting opponents, etc.
These failures and mistakes occur due to the nature of AI, which relies on a large database that it pulls from to inform decision-making[9]. However, if the database is too small, lacks particular or rare scenarios, or is poisoned, then the AI will not be able to competently respond to a situation.[10] Due to the chaotic nature of the real world and especially of conflict scenarios, the databases for AI programs are likely insufficient.[11] AI programs also have an element of randomness that is sometimes irrational to human beings and will result in poor decision-making or confuse human beings with their decisions[12]. This element of randomness has the chance of making decisions that can escalate a situation or confuse opposing leadership and thus escalate the situation.[13] AI programs also lack “common sense”, while humans may be able to understand a command such as defend an area from opposing forces during a period of peace meaning to stand guard but not start a conflict[14]. AI programs may attack enemy positions if given the same order due to them believing it is the best way to defend the position because they lack nuance or common sense.[15] AI programs also require a large amount of training on their database to learn competent decision-making in the situation they are intended for.[16] While some countries like the U.S. may be aware of these risks and make it a priority to limit the chance of AWS failures.[17] Many countries around the world may feel the need to field or use AWS before they are ready or sufficiently tested due to their possible benefits.[18]
How Can an AWS Failure Lead to Unintentional Conflicts
There are multiple reasons why these AWS failures or mistakes may lead to conflict or escalation, one of which is the novelty of AWS. Political, military, and diplomatic leaders are used to a world of engaging with other humans, and conflict or de-escalation is in part about being able to read and understand the opposing side followed by suitable actions.[19] However, political, military, and diplomatic leaders lack the skills or experience to engage with AI and AWS. There is also the issue that AI and AWS may lack the understood norms and guidance that humans may share.[20] This lack of understanding or experience in human-AI engagement, especially during tense periods, conflicts, or de-escalation, may lead to either the humans or AWS undertaking actions that may escalate or create conflict.
Another possible reason that AWS failures, glitches, or mistakes may cause accidental escalation or conflict is the speed at which AWS can make decisions or actions. During periods of increased tensions or conflict, leaders have limited time to make decisions. By limiting the amount of time that leaders have to make decisions, leaders are limited in the ability to collect information, explore options, and make carefully thought decisions. However, AWS can perform actions and make decisions much faster than humans are capable of doing. The speed at which AWS can perform or make decisions significantly shortens the amount of time that opposing leaders have to make decisions, especially during periods of conflicts or high tension. If an AWS were to one day fail and perform an action that may escalate tensions or cause the outbreak of conflict, the amount of time that opposing human leaders have to respond is significantly shortened.[21] The shortening of time may cause opposing leaders to make decisions that further escalate the situation or cause conflict to break out due to poor decision-making.[22]
The major reason that AWS failures, glitches, or mistakes may cause accidental escalation or conflict is due to the issue of attribution and ambiguity. Attribution is when an entity can discern who performed an action if the action was intentional, and why the action was performed.[23] Attribution is extremely important for the decision-making process of leaders, especially during times of conflict or tension as attribution allows leaders to understand an action and guide a response. But if an AWS were to fail or malfunction and perform an action that escalates a situation, it is difficult for opposing leaders to attribute the action to an intentional order by human leaders or to an AWS failure.[24] It is difficult to attribute AWS actions due to the novelty of AWS and leaders lacking experience or knowledge in engaging with AWS[25]. It is also difficult to attribute AWS’s actions because of their autonomous nature and the inability of foreign leaders to tell what orders AWS received and how they interpreted them[26]. The lack of attribution of the actions of AWS to either an AWS failure or an intentional order creates ambiguity in contested situations[27]. The ambiguity in these contested situations greatly increases the chance of misunderstanding, miscommunication, and thus poor decision-making that generates more conflict. The chance of an AWS failure creating accidental conflicts is then further compounded by the novelty of interacting with AWS and the decreased amount of time to make decisions due to the speed of AWS, increasing the chance of poor decision-making by human leaders.
Recommendation:
To limit the chance of an AWS failure, glitch, malfunction, or mistake leading to an accidental escalation or conflict, I recommend the U.S. should create and promote the norm of publicizing instances of AWS failures, glitches, malfunctions, and mistakes as well as their locations as a confidence-building measure. Publicizing instances of AWS failures and their locations will limit the chance of accidental conflict in two ways. Publicizing the locations where AWS is being used, allows for foreign or opposing leaders to prepare and set protocols for engaging with AWS. By allowing foreign or opposing leaders to prepare and set protocols for AWS engagement, you are giving foreign leaders more time to make decisions for when an AWS interaction occurs. Secondly, you are allowing foreign leaders to learn about AWS and decrease their novelty. Both of these outcomes will decrease the chance of poor decision-making in the event of an AWS failure that results in accidental escalation.
By publicizing instances of AWS failures, glitches, malfunctions, and mistakes you will be partially removing the ambiguity of AWS interactions. For example, if the U.S. had an AWS that was well known for having failures due to publicizing them, then the AWS was to fail and perform an action that could lead to escalation with China, such as firing on a Chinese naval vessel. The Chinese leadership will know that the U.S.’s AWS is prone to failures, as well as the firing on a Chinese naval vessel is uncharacteristic of the United States. These two factors combined will remove much of the ambiguity of the interaction by pointing heavily that the firing on the vessel was due to an AWS failure rather than an intentional U.S. attack. If a country were to follow the policy of publicizing AWS failures but possessed an AWS that rarely if ever failed. Then a scenario were to occur where said county’s AWS was to attack the opposing forces of another country. The ambiguity of the attack would still be partially removed as the leaders of the country that was attacked would know that this attack was unlikely to be an AWS failure and more likely an intentional order from opposing human leaders.
Even if there was a scenario of an AWS that had some failures but not many, limiting the ability of foreign leaders or intelligence agencies to confidently discern if an AWS action was due to a failure or an intentional order. By publicizing what few times the AWS failed, you will be injecting the possibility that the AWS’s actions were caused by an AWS failure and not an intentional order into the decision-making process of the opposing leader. The addition of the possibility of an AWS failure into the decision-making process of opposing leaders would give pause to those leaders and decrease the chance of poor decision-making on their part. The publicizing of AWS failures vastly decreases the chance of an AWS failure leading to an accidental conflict or escalation due to poor decision-making, by either removing the ambiguity of AWS actions or giving pause to opposing leaders in their decision-making process. The publicizing of AWS failures will also remove the novelty of AWS, allowing leaders to become familiar with them and plan for possible engagements again decreasing the chance of accidental conflicts.
There are a few possible drawbacks to this policy such as the publication of possible confidential capabilities, the erosion of trust in AWS, using this policy as an excuse to perform actions, and allowing for possibly hostile nations to locate and destroy AWS. However, the U.S. has taken up other confidence-building measures that could have exposed our capabilities or allowed hostile nations to locate and destroy strategic assets. During the Cold War, the U.S. and Soviet Union, followed by the Russian Federation, signed multiple nuclear and ballistic missile arms treaties limiting each other’s weapons as well as capabilities.[28] To build confidence with the Soviet Union and later Russia that we were abiding by the treaty, we allowed them to monitor our strategic nuclear assets and we did the same with them.[29]
The publicizing of AWS failures is also unlikely to reveal our capabilities or lack thereof. Unlike a hardware failure such as an error in the F-35 that compromises effectiveness, which would require billions of dollars and years to solve then implement the solution across the F-35 fleet. Most LAWS failures would be due to software issues which would be significantly less costly and time-consuming to fix, then upload the new software into the AWS fleet. The U.S. could also wait to publicize the AWS failure or location until the AWS location or error has been fixed, limiting the chance of exposing our capabilities or exposing us to an attack.
There is a real possibility countries may use the publicizing of AWS failures as a way to cover for covert or overt actions. For example, Russia used AWS to attack a Finnish base and then claimed it was an AWS failure to avoid responsibility. However, countries may still do such actions and claim AWS failure even if we do not adopt this policy. Countries are also incentivized to not abuse or mislead instances of AWS failures to cover intentional actions as it may lead to a “boy who cried wolf” effect where an instance of actual AWS failure is not believed and thus results in unwanted conflict.
The publicizing of AWS failures may also build public trust in AWS rather than erode it by showing our continuous improvement of the weapon systems and their capabilities. Finally, if we were to publicize our AWS failures, we would alert all other countries with AWS of the possible failure that could occur on their system, limiting the possibility of their AWS failing and leading to conflict globally.
Conclusion: AI-lethal autonomous weapon systems are likely to begin seeing the battlefield or widespread adoption in the future. However, due to their novelty, speed, and ambiguity, there is a major risk of accidental escalation or conflict caused by an AWS failure. By publicizing AWS failures and locations, we decrease the novelty, speed, and ambiguity of AWS, thus decreasing the chance of poor decision-making as well as escalation or conflict. While there are some issues with the publication of AWS failures and locations such as releasing confidential information or allowing adversaries to track and target AWS. The U.S. has shown in the past through nuclear arms treaties that it is willing to make this sacrifice.
[1] Bendett Sam, 2023, “Roles and Implications of AI in the Russian-Ukrainian Conflict.” Russia Matters, https://www.russiamatters.org/analysis/roles-and-implications-ai-russian-ukrainian-conflict.
[2] Klare, Michael T, 2019, “Autonomous Weapons Systems and the Laws of War.” Arms Control Association, https://www.armscontrol.org/act/2019-03/features/autonomous-weapons-systems-laws-war.
[3] Department of Defense, June 27, 2023, “Data, Analytics, and Artificial Intelligence Adoption Strategy, Accelerating Decision Advantage” DoD, https://media.defense.gov/2023/Nov/02/2003333300/-1/-1/1/DOD_DATA_ANALYTICS_AI_ADOPTION_STRATEGY.PDF
[4] Klare, Michael T, “Autonomous Weapons Systems…”
[5] Department of Defense, January 25, 2023, “DoD Directive 3000.09 Autonomy in Weapon Systems” DoD, https://www.esd.whs.mil/portals/54/documents/dd/issuances/dodd/300009p.pdf
[6] Defense Science Board, 2016, “Report of the Defense Science Board Summer Study on Autonomy.” Summer Study on Autonomy, https://irp.fas.org/agency/dod/dsb/autonomy-ss.pdf
[7] Defense Science Board “Report of the…”
[8] Defense Science Board “Report of the…”
[9] Horowitz, Michael c, and Paul Scharre, 2021, “AI and International Security Risks and Confidence-Building Measures.” Center for New American Security https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/AI-and-International-Stability-Risks-and-Confidence-Building-Measures.pdf.
[10] Horowitz and Scharre “AI and International Security…”
[11] Horowitz and Scharre “AI and International Security…”
[12] Horowitz and Scharre “AI and International Security…”
[13] Horowitz and Scharre “AI and International Security…”
[14] Horowitz and Scharre “AI and International Security…”
[15] Horowitz and Scharre “AI and International Security…”
[16] Horowitz and Scharre “AI and International Security…”
[17] Defense Science Board “Report of the…”
[18] Klare, Michael T, “Autonomous Weapons Systems…”
[19] Horowitz and Scharre “AI and International Security…”
[20] Horowitz and Scharre “AI and International Security…”
[21] Wong, Yuna Huh, John Yurchak, Robert W. Button, Aaron B. Frank, Burgess Laird, Osonde A. Osoba, Randall Steeb, Benjamin N. Harris, and Sebastian Joon Bae, 2020, “Deterrence in the Age of Thinking Machines. Santa Monica, CA: RAND Corporation, https://www.rand.org/pubs/research_reports/RR2797.html.
[22] Wong, Yurchak, Button, Frank, Laird, Osoba, Steeb, Harris, and Bae, “Deterrence in the…”
[23]Lynch, Justin, and Emma Morrison. 2023. “Deterrence Through AI-Enabled Detection and Attribution.” Johns Hopkins SAIS. https://sais.jhu.edu/kissinger/programs-and-projects/kissinger-center-papers/deterrence-through-ai-enabled-detection-and-attribution.
[24]Horowitz and Scharre “AI and International Security…”
[25]Horowitz and Scharre “AI and International Security…”
[26]Horowitz and Scharre “AI and International Security…”
[27]Horowitz and Scharre “AI and International Security…”
[28] Congressional Research Service. December 23, 2011. “Monitoring and Verification in Arms Control”. Congressional Research Service. https://crsreports.congress.gov/product/pdf/R/R41201/9#:~:text=The%20United%20States%20and%20Soviet,activities%20of%20other%20participating%20states.
[29] Congressional Research Service “Monitoring and Verification in Arms Control”
