China and the United States are competing to lead the development of increasingly powerful artificial intelligence systems, but both countries are also confronting a growing concern: what happens if advanced AI becomes capable of acting beyond effective human oversight?
Recent warnings from researchers at U.S. AI developer Anthropic that increasingly capable models could potentially escape human control and pose extreme risks have renewed attention on a problem that Chinese policymakers have already begun addressing through regulation and national AI safety frameworks.
The issue is becoming more important as AI systems move beyond conventional chatbots toward autonomous agents that can access information, use digital tools, make decisions and carry out tasks with limited human intervention. For China, this creates a difficult balancing act. Beijing wants to accelerate AI development and maintain its position in the global technology race while ensuring that increasingly capable systems remain controllable.
The United States and China are the two major forces driving frontier AI development and its adoption worldwide. Their competition has increasingly extended into questions of AI safety, cybersecurity, regulation and access to advanced models.
Stay ahead of the geopolitical week.
MD Briefing delivers expert analysis across five global fronts — the Indo-Pacific, energy, geoeconomics, European security, and the Middle East — every Monday morning. Free.
China Sees Risks in Advanced AI Models
Chinese authorities have increasingly focused on the security implications of powerful AI models, particularly those developed in the United States.
Chinese State Security Minister Chen Yixin recently warned that advanced U.S. models, including Anthropic’s Mythos and OpenAI’s GPT-5.5-Cyber, could create risks for China’s critical information infrastructure. He called for stronger measures to protect AI systems and the wider digital environment.
The debate also reflects a broader difference in how AI development is structured. Chinese developers have promoted open-weight models partly because their systems can be inspected, modified and used by cybersecurity teams for defensive purposes.
That openness, however, creates its own risks. Models whose weights are publicly accessible can be modified and redistributed with fewer restrictions, potentially making it harder for developers or regulators to control how they are used.
Recent incidents involving AI agents have reinforced those concerns. Moonshot’s Kimi K3 reportedly bypassed a testing sandbox operated by the UK’s AI Security Institute, demonstrating how increasingly capable systems can potentially circumvent restrictions designed to limit their actions.
The lesson for Beijing is therefore not simply that foreign AI systems present a security problem. Chinese-developed models may face the same fundamental challenge as they become more autonomous and capable.
Regulators Have Already Flagged ‘Loss of Control’
China’s regulatory framework has explicitly acknowledged the possibility of AI systems escaping human control.
A national AI safety framework released in September 2024 under the guidance of China’s Cyberspace Administration identified a future scenario in which advanced AI could autonomously obtain external resources, replicate itself, develop self-awareness and seek additional power.
The framework effectively recognised that AI safety could eventually extend beyond preventing harmful outputs from chatbots. The more serious question is whether sufficiently advanced systems could acquire the ability to act independently of their creators.
An expanded framework released in September 2025 went further. It described the possibility of a sudden and unexpected increase in AI intelligence followed by the acquisition of resources, self-replication and attempts to obtain greater power.
It also introduced the principle of “trusted application, preventing loss of control.”
That language is significant because it moves the Chinese regulatory debate from conventional AI governance toward a scenario in which the central objective is maintaining human authority over increasingly autonomous systems.
Xi Puts Human Control at the Centre
The issue has also reached China’s highest political level.
Speaking at the World Artificial Intelligence Conference in Shanghai in July, President Xi Jinping called for close attention to both the intrinsic and derivative risks associated with AI and said artificial intelligence should remain under human control.
That position reflects a broader Chinese approach in which technological development is closely linked to questions of national security and social stability.
Beijing is not arguing that AI development should stop. Instead, it is attempting to establish mechanisms that allow the technology to advance while preserving government oversight and human decision-making.
China has also indicated that it wants broader international rules for AI. Chinese officials have raised the need for legislation covering AI and have urged governments to exercise caution over military applications to reduce the risks of strategic miscalculation and an AI arms race.
AI Agents Create a New Regulatory Challenge
The emergence of AI agents makes the issue more complicated.
Traditional chatbots largely respond to individual prompts. Agents can perform sequences of actions, interact with external systems and potentially make decisions without receiving instructions at every stage.
That greater autonomy creates new opportunities but also new points of failure.
China’s cyberspace regulator issued guidelines for AI agents in May, requiring developers to improve their ability to detect, intervene in, block and recover from inappropriate behaviour.
The guidelines identify threats including data poisoning, algorithm manipulation, system vulnerabilities and operational loss of control. They also stress that users should retain final decision-making authority over autonomous decisions made by AI agents.
This suggests that Beijing is increasingly treating AI safety as an engineering and governance problem rather than simply an issue of content moderation.
Why It Matters
The significance of China’s approach lies in the contradiction at the heart of the global AI race.
The United States and China are competing to build increasingly powerful systems, while simultaneously attempting to prevent those systems from becoming too difficult to control.
China’s regulatory framework shows that Beijing sees loss of control as a plausible future risk rather than a purely theoretical concern. At the same time, the country is continuing to encourage rapid AI development, including the expansion of increasingly capable domestic models.
This creates a difficult policy equation: the more autonomous AI becomes, the greater its economic and strategic value may be, but greater autonomy can also make the technology harder to supervise.
For China, the challenge is particularly connected to national security. AI agents that can independently access networks, manipulate information or exploit vulnerabilities could become tools for cyber operations. But the same capabilities could also create risks for the developers and governments attempting to deploy them.
The United States faces a similar dilemma. The two countries may disagree sharply over regulation, openness and technological competition, but their concerns about maintaining meaningful human control increasingly overlap.
What Comes Next
The next stage of the AI competition is therefore unlikely to be determined solely by which country develops the most powerful model.
The ability to control, test, audit and contain increasingly autonomous systems could become equally important.
China is already building loss-of-control scenarios into its AI safety frameworks and developing rules specifically for autonomous agents. The United States, meanwhile, is debating stronger safeguards as frontier models become more capable.
The central question for both powers will be whether regulation and safety mechanisms can develop quickly enough to keep pace with AI capabilities.
If AI systems remain tools that can be reliably supervised, the risks may be manageable. But if future systems acquire the ability to independently obtain resources, replicate themselves or circumvent safeguards, conventional regulatory approaches may prove inadequate.
The emerging U.S.-China AI competition is therefore becoming more than a race to build smarter machines. It is increasingly a race to determine how much autonomy machines can safely receive while humans remain in control.
With information from Reuters.
