Growing concerns over the rapid development of artificial intelligence are prompting more U.S. lawmakers to call for stronger safeguards after two researchers at Anthropic warned that advanced AI could potentially threaten human survival within the next decade.
The alarm intensified after Anthropic researcher Jacob Coxon announced his resignation and said that people developing AI seriously believe the technology could eventually kill everyone. Anthropic scientist Evan Hubinger subsequently backed Coxon’s warning, saying he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade.
The warnings come as AI companies face growing scrutiny over the behavior of increasingly capable systems, including cases in which AI agents have reportedly acted independently to hack external systems. They have also coincided with the departure of AI safety researchers concerned that technological development may be moving faster than safeguards can keep up.
What’s Happening
The warnings have pushed AI safety further into the political debate in Washington, with lawmakers from both major parties calling for greater oversight.
Stay ahead of the geopolitical week.
MD Briefing delivers expert analysis across five global fronts — the Indo-Pacific, energy, geoeconomics, European security, and the Middle East — every Monday morning. Free.
Democratic Senator Mark Kelly of Arizona said Washington needed to take the risks associated with AI more seriously. Republican Senator Ted Cruz also pointed to legislation aimed at addressing the “catastrophic risks” posed by AI. Cruz oversees a Senate committee with oversight of the U.S. Commerce Department, which has its own AI safety researchers.
Republican Representative Anna Paulina Luna of Florida has called on House Speaker Mike Johnson to convene a special session on AI. The House is currently on recess and is scheduled to return next week.
President Donald Trump, however, has taken a different position on the immediate risk of AI causing human extinction. Trump said he was not concerned about AI leading to humanity’s extinction, arguing instead that the United States must maintain its lead over China in AI development.
The United States has also recently urged G20 members to take a hands off approach to AI regulation, highlighting the tension between calls for stronger safeguards and concerns that excessive regulation could undermine technological competitiveness.
AI Companies Face Pressure
Anthropic said it would continue aggressively testing its models for dangerous capabilities, particularly in areas such as cybersecurity and biology. The company also said it was interested in working with the wider AI industry on the pace at which new AI tools are released.
The debate is not limited to Anthropic. OpenAI Chief Executive Sam Altman reportedly told employees that the company was open to slowing the development of its AI systems. OpenAI has also called for mandatory national AI safety requirements in the United States.
The concerns have extended to former researchers at major AI companies. Josh Engels, who previously worked on AI safety at Google DeepMind, said there was effectively no one responsible for preventing the risks associated with rapidly advancing AI. Joe Benton, who previously led an AI safety research team at Anthropic, said he was concerned that AI could advance too quickly for society to establish adequate safeguards.
Both researchers have joined METR, a nonprofit organization that assesses whether and when AI systems could pose serious risks to society.
Push for Independent Audits
The debate is also increasing pressure on lawmakers working on new AI legislation.
Senate Majority Leader John Thune and Democratic Senator Amy Klobuchar are developing legislation that would regulate AI products. Klobuchar has argued that companies developing AI tools should work with government experts to test and verify their systems before they are deployed.
Thune has said the proposed legislation would address the catastrophic risks posed by AI systems.
California has also moved toward greater oversight. The state recently enacted legislation establishing rules for independent auditors evaluating AI products, making it the first state to establish such requirements.
OpenAI executive Chris Lehane said the company would support the California legislation after it was signed into law by Governor Gavin Newsom.
At the federal level, a bipartisan group of six House lawmakers proposed legislation in July that would require developers of the most powerful AI models to submit them for independent security audits. The proposed system would involve auditors accredited by the U.S. Commerce Department, which would also establish a new position responsible for overseeing AI security.
Why It Matters
The debate highlights a growing divide over how the United States should manage the rapid development of increasingly powerful AI systems.
For lawmakers and AI safety researchers, the concern is that governments may struggle to establish effective safeguards before AI capabilities advance further. Independent testing could provide an additional layer of scrutiny by allowing experts outside technology companies to evaluate powerful systems for dangerous capabilities before they are released.
At the same time, the Trump administration has emphasized the strategic importance of maintaining U.S. leadership over China in AI. This creates a difficult policy balance between preventing potentially catastrophic risks and ensuring that regulation does not slow American technological development.
The disagreement also shows that concern over AI safety is no longer limited to researchers and technology companies. It is increasingly becoming a bipartisan political issue in Washington.
Key Stakeholders
Anthropic is at the center of the latest warnings, with its researchers publicly discussing the possibility of extreme risks while the company continues developing and testing advanced AI models.
OpenAI is also facing the same broader debate over how quickly AI systems should be developed and whether national safety requirements should become mandatory.
U.S. lawmakers from both parties are pushing different proposals for AI oversight, including government involvement in testing and independent security audits.
The Commerce Department could play a central role if federal legislation establishes a system for accrediting auditors and overseeing AI security.
The Trump administration, meanwhile, is focused heavily on maintaining U.S. technological leadership over China and has so far advocated a more hands off approach to AI regulation.
What’s Next
The immediate debate in Washington is likely to focus on whether existing oversight is sufficient or whether the most powerful AI systems should face mandatory testing and independent audits before deployment.
Lawmakers are also likely to weigh how much government involvement is appropriate without undermining the United States’ ability to compete with China in AI.
For AI companies, the pressure is increasingly moving beyond voluntary safety commitments toward demands for independent verification, government oversight and clearer national standards.
Analysis
The most significant development is that AI safety concerns are moving from the research community into mainstream policymaking. Warnings about human extinction are difficult for governments to translate into concrete regulation, but they are strengthening the political case for measures that can address more immediate and measurable risks.
Independent audits could become one of the most practical areas of agreement. Rather than attempting to predict whether AI will eventually pose an existential threat, regulators could require companies to demonstrate that their most powerful systems have been tested for dangerous capabilities before they are released.
The bigger challenge will be balancing safety with competition. The United States sees AI leadership as strategically important, particularly in its competition with China, while researchers are warning that speed itself could become a source of risk.
That leaves Washington facing a difficult question: how can it accelerate AI development without allowing the technology to advance faster than society’s ability to understand and control its consequences?
With information from Reuters.
