AI Error Nearly Led to US Attack on Chinese Ship, Risking Major War
An intelligence report circulating within the United States military last spring nearly triggered a serious incident with China. The report claimed that a Chinese vessel sailing in the Middle East was carrying components for a nuclear weapons programme.
This information immediately prompted US military action. According to four sources familiar with the incident, the US military had prepared plans to intercept the ship. Two sources, as quoted by CNN, stated that armed military personnel were prepared to board the vessel, while US military aircraft were already airborne.
However, just before the operation was executed, US officials conducted a more in-scale verification of the intelligence report. The investigation revealed that the report had been produced with the assistance of artificial intelligence (AI).
More shockingly, the chatbot used by an analyst during the process had misidentified the materials being carried by the Chinese ship. According to one source, the report was “entirely incorrect”. However, the same source noted that the error “almost started a war”.
If the US had proceeded with the operation against the Chinese vessel, the situation could have escalated into an armed conflict between the world’s two greatest military powers.
This incident occurred as the US military and intelligence community are aggressively integrating AI into almost all areas of their operations. The technology is being used to analyse vast volumes of raw intelligence, assist in determining attack targets, and handle administrative tasks such as budget management, logistics, and supply chains.
However, the case of the Chinese ship demonstrates the massive risks involved when relatively new technology, which is not yet fully understood, is used to determine targets during wartime. Analysts have long warned that using AI with poor or corrupted data can lead to miscalculations resulting in catastrophe. In extreme situations, such errors could lead the US to fire upon a Chinese ship simply because the information generated by the system was erroneous.
In this instance, the analyst entered various pieces of intelligence regarding the ship’s manifest into a chatbot. The initial information originated from the US Special Operations Command Pacific, based in Hawaii. It remains unclear whether the chatbot used was a commercial service available to the public or a product developed specifically by the US government.
A former senior US official familiar with the AI systems used by military and intelligence analysts stated, “Those internal tools are largely just polished copies of commercial products made to look better.”
The chatbot then synthesised open-source intelligence with classified signals intelligence stored within government systems. From this combination, the system generated conclusions regarding the materials being transported.
The analyst then used AI again to compile the findings into a standardised intelligence report format. Such reports are typically trusted by military officials and subsequently disseminated to relevant parties.
The drive to accelerate AI usage in the military is based on the assumption that the technology can assist in making battlefield decisions more rapidly—for example, determining which targets to strike or deciding which military assets should be moved to specific locations.
US officials also believe that Washington must not fall behind in the development and implementation of AI, particularly if it must one day face China or other nations capable of using the technology to move a step ahead of the US.
In January, US Secretary of Defense Pete Hegseth launched a strategy titled the “Artificial Intelligence Acceleration Strategy” to speed up AI adoption within the military environment. “We will create space for experimentation, remove bureaucratic hurdles, centralised our investments, and demonstrate the necessary implementation approach to ensure we lead in military AI,” Hegseth said in a speech announcing the strategy.
The strategy also encourages widespread AI use throughout the military. The Department of Defense was instructed to provide AI through several programmes with the aim of “democratising experimentation and AI transformation across the Department by placing America’s leading AI models directly into the hands of our three million civilian and military personnel, across all levels of classification.”
However, the implementation of AI within the US government is currently decentralised. Several US officials aware of the process stated that various government agencies use different tools, with commands and security standards that are not uniform. Furthermore, the US does not yet have a single standard for how AI-generated information should be verified. The proliferation of various AI systems used across different military and intelligence branches means that the reliability and capability of each system can vary significantly.
In recent weeks, policymakers in Washington have been debating the risks of AI intensely. This debate was triggered by a series of warnings from Silicon Valley engineers and tech CEOs regarding the possibility of AI evolving beyond human control, with consequences that could potentially threaten civilisation.
However, the Chinese ship incident demonstrates a different, more immediate threat. It is not just about AI one day escaping human control, but rather the human element.