A recent wave of AI security incidents has intensified an argument that was already reshaping the industry: should companies slow the development of increasingly capable models, or would restraint simply hand strategic and economic advantages to competitors?
The controversy began with claims that Israeli firm Irregular helped build evaluations behind hacks involving models from OpenAI, Anthropic, and Meta. Critics argue that the incidents were less evidence of autonomous “rogue agents” than of poorly scoped capture-the-flag tasks and models given overly broad access to the internet. Some have described the episode as a “pacing provocation”—a staged or amplified failure intended to demonstrate that advanced AI is uncontrollable and justify slowing the field.
Those allegations remain contested. But the underlying question is real. Modern AI systems can browse websites, write and execute code, call external tools, and pursue multistep objectives. An evaluation designed to test those abilities can itself create dangerous permissions. If a model is allowed to explore an untrusted environment without carefully defined limits, researchers may end up measuring the consequences of their test design as much as the model’s capabilities.
That distinction matters. A system that exploits a poorly configured benchmark is not necessarily an autonomous agent capable of escaping human control. Yet such failures can still reveal how quickly a model may find unintended pathways through software, organizations, and information networks. The practical challenge is to separate genuine capability from artifacts of an evaluation—and to build tests that are rigorous without becoming attack surfaces.
The political response has been sharply divided. President Donald Trump criticized warnings about catastrophic AI risk as a “hoax” in a call with Nvidia CEO Jensen Huang, who has said that existential-risk claims are not grounded in science. Trump also attacked Anthropic CEO Dario Amodei’s call for a slower development pace, blaming opposition to new data centers on what he called a “SICK conspiracy” and arguing that the country needs a “High IQ!” president as its main safeguard.
Markets nevertheless reacted uneasily. Nvidia shares fell about 3 percent and AMD shares 4.4 percent during the episode, reflecting a broader tension: investors expect AI companies to move quickly, but they also worry that accidents, regulation, or public backlash could disrupt the infrastructure boom. Huang’s argument that “whoever wins AI, wins” captures the strategic logic. If each company or country believes that slowing down unilaterally will leave it vulnerable, collective restraint becomes a classic prisoner’s dilemma.
The leading laboratories appear reluctant to accept that restraint means stopping. OpenAI CEO Sam Altman has said that pacing should not mean abandoning progress, while supporting federal rules and independent auditors. OpenAI has also described preparing safety cases before large reinforcement-learning runs. Anthropic, OpenAI, and Google were reportedly discussing a standards organization before Amodei published his letter calling for stronger controls.
Amodei’s dilemma is particularly acute in relation to China. A shared “speed limit” could reduce accident risk, but it could also weaken the United States’ military and economic position if China does not participate. Altman, Google DeepMind CEO Demis Hassabis, and Elon Musk have expressed support for Amodei’s concerns, while White House AI adviser David Sacks has told laboratories to stop acting as though they need permission to proceed. At the same time, venture capitalist Garry Tan has argued against aggressive action to prevent Chinese model distillation, despite a new advisory from the NSA, CISA, and FBI warning about the risks of stolen or replicated AI capabilities.
Microsoft has proposed a middle path: faster development accompanied by stronger oversight, including “embedded evaluators” inside the development process. CEO Satya Nadella has said control should not be concentrated in a handful of companies. Microsoft has also issued a code of conduct emphasizing that people matter more than AI, rejecting claims that current models possess consciousness, personhood, or welfare interests. The document implicitly challenges research programs focused on possible model sentience and rejects a race to build an all-purpose superintelligence.
International competition makes coordination even harder. China rejected Amodei’s appeal for restraint, with its Foreign Ministry criticizing what it called fearmongering and confrontation before a planned Trump-Xi meeting. Analysts have pointed out that a country in second place has little incentive to stop there. Instead, Xi Jinping has promoted an open-source AI community among BRICS and other developing countries, with safety receiving less attention than access and technological sovereignty.
The US Congress is also unlikely to impose a broad pause soon. House Speaker Mike Johnson has warned against an emergency moratorium, while Democrats led by Hakeem Jeffries have scheduled discussions about slowing development. King Charles III has separately invited leaders from Nvidia, Google DeepMind, OpenAI, and Anthropic to Dumfries House to discuss how AI might benefit society. The meeting follows the hacking incident, which has since been attributed by some reports to a misconfigured evaluation.
The capabilities driving the debate are not theoretical. Anthropic’s Claude Fable 5.1 reportedly solved Thomas Urquhart’s 370-year-old Cyphral Distich in 44 minutes after recognizing that the cipher’s key was the book itself. The model used numerical references to index words in one of Urquhart’s works, revealing a Royalist prayer for Charles II, and then made substantial progress on a related cipher. Anthropic is now applying similar language-analysis capabilities commercially through Claude for Financial Advisors, integrated with firms including BlackRock, Charles Schwab, Addepar, and Envestnet.
Meanwhile, the hardware race continues at full speed. Early verified results for Nvidia’s Vera Rubin NVL72 platform reportedly show as much as 67 times the inference throughput per total cost of ownership of the company’s GB300 system, and roughly seven times the tokens per megawatt—well above Huang’s public claim of a threefold improvement. Such numbers illustrate why companies have little appetite for voluntary delay: better inference economics can quickly translate into cheaper services, larger workloads, and a competitive advantage.
The infrastructure buildout is also becoming a political negotiation. Amazon, Microsoft, and Oracle are offering municipalities financial incentives and other benefits as they seek approval for data centers. At the same time, technology companies are siding with consumers against utilities that want ratepayers to help fund the additional generation and grid upgrades required by AI facilities. The dispute highlights a central economic question: who should pay for the energy and infrastructure needed to support private AI profits?
The effects extend beyond software. XPeng has announced what it calls the first automated humanoid-robot production line, in which robots manufacture robots; its first unit, named Iron, reportedly left the line without human assistance. In the military sphere, Air Force Secretary Troy Meink has acknowledged that the United States has weapons in orbit, arguing that disclosure can strengthen deterrence. The Department of War has also issued a legal waiver allowing personnel to provide classified information about unidentified anomalous phenomena to the PURSUE effort despite nondisclosure agreements.
As companies invest in automation, lawmakers are beginning to focus on who receives the gains. Senators Bernie Sanders and Mark Takano are reintroducing the Thirty-Two Hour Workweek Act, backed by labor organizations that argue productivity improvements should benefit workers rather than a small group of billionaires. The debate over AI pacing is therefore inseparable from the debate over AI distribution: even a safe and highly productive technology can deepen inequality if its rewards are narrowly captured.
The laboratories themselves are not behaving as though a slowdown is imminent. Anthropic has told shareholders that it expects adjusted operating income to remain positive for a second consecutive quarter, with gross margins above 80 percent. The company has reportedly considered an October Nasdaq listing at a valuation as high as $2 trillion, while Altman has said OpenAI does not plan to go public amid the current controversy.
The central uncertainty is not whether AI development will stop. It is whether safety practices, public institutions, and labor and energy policies can evolve quickly enough to keep pace. The recent hacks may not prove that AI systems are uncontrollable, but they do show how easily ambitious experiments can create unintended risks. The challenge now is to replace slogans about acceleration or doom with measurable standards: carefully bounded evaluations, independent audits, transparent incident reporting, and rules that distribute both the benefits and costs of increasingly capable machines.