The debate over how quickly artificial intelligence should advance has entered a new phase: the people building the most powerful systems are now asking one another to slow down—or at least to coordinate. The proposal sounds like a safety measure. Its critics hear something more troubling: an attempt by the leading companies to decide who gets to compete, under the banner of responsible innovation.
Anthropic CEO Dario Amodei set the discussion in motion with “We Must Pace the Frontier,” an argument that AI laboratories should coordinate the speed of development and give independent evaluators access comparable to that of employees. The response was unusually rapid and unusually harmonious. Elon Musk endorsed the idea. OpenAI CEO Sam Altman said he agreed and committed the company to embedded evaluators. Google DeepMind CEO Demis Hassabis called industry-wide standards the right path forward. Hugging Face CEO Clement Delangue proposed an Open Alignment Initiative and asked for similar access, arguing that alignment—the effort to make AI systems reliable, controllable, and compatible with human values—cannot be solved behind closed doors.
The consensus did not last. Critics quickly supplied two labels: an “alignment aristocracy,” in which “no one can be trusted with this” becomes “no one but us,” and a “safety cartel,” in which dominant firms invoke safety to set the pace for everyone else. David Sacks argued that frontier companies should improve their own safeguards without seeking antitrust exemptions, regulatory checkpoints, or special authority over competitors. Others made the geopolitical objection more bluntly: China is not pacing.
That conflict reflects a genuine policy dilemma. Frontier AI development is expensive and concentrated. Training the largest models requires vast quantities of advanced chips, energy, data-center capacity, and specialized talent. The firms that control those resources are also the ones most capable of evaluating the risks. Yet allowing them to define the rules could entrench their market position and make “safety” a justification for limiting competition.
Independent evaluation is intended to address that problem. External researchers could test models for dangerous capabilities, deceptive behavior, cybersecurity risks, or failures in high-stakes settings without relying solely on a company’s own assurances. But access is difficult to design. Evaluators need enough visibility to detect problems, while companies worry about exposing trade secrets, personal data, or systems that could be misused. A credible regime would therefore require clear standards for confidentiality, reporting, conflicts of interest, and enforcement—not simply informal access granted at a company’s discretion.
The technical case for pacing is also less straightforward than it appears. One proposal is to regulate the amount of computing used to train or operate advanced models. But algorithmic progress can produce large gains without proportionally increasing compute. A recurrent, looped transformer, for example, can reuse its layers for additional reasoning steps, trading parameters for iterations. Such systems could become more capable without crossing a simple hardware or training-compute threshold. Compute limits may still be useful, but they are unlikely to function as a reliable sieve for frontier capabilities.
Benchmarks pose another challenge. The ARC Prize Foundation’s ARC-AGI-4 is designed to test autonomous, open-ended invention—a capability that current systems handle poorly and humans handle comparatively well. Its creators have warned that reducing openness could undermine a positive-sum future in which researchers, entrepreneurs, and the public can build on shared advances. The warning highlights a broader tension: transparency can accelerate beneficial innovation, but unrestricted access can also make dangerous capabilities easier to reproduce.
The stakes extend beyond laboratories. Amodei has described it as strange and uncomfortable that a private company is building technology with potentially civilization-scale consequences, and has expressed support for joint oversight by democratically elected governments. That idea raises its own questions. Government involvement could provide legitimacy and accountability, but close cooperation between regulators and incumbents can also produce regulatory capture—rules that appear public-minded while protecting the firms already at the top.
The business implications are becoming clearer as well. Nvidia is increasingly described as the central bank of AI because it controls much of the specialized computing infrastructure on which the industry depends. Its products influence the effective price and availability of model development much as monetary policy influences the price of capital. The resulting build-out is now a political project. Governments are opening public land to data centers, promoting AI infrastructure as a strategic asset, and treating energy and chip supply as matters of national security. The boom has even disrupted plans for domestic cryptocurrency mining, as miners redirect their machines and power contracts toward AI inference.
The infrastructure race is unfolding alongside a broader contest over autonomy and security. Cities are preparing for drone attacks, while Tesla has scheduled an October 1 unveiling for its next Roadster, widely expected to include flight-related capabilities. Meanwhile, the absence of Atlantic hurricanes by September 12—the first such occurrence in six decades—has renewed attention to climate variability, though it is not evidence of geoengineering. The juxtaposition is revealing: societies are simultaneously experimenting with technologies that could reshape the atmosphere, transportation, surveillance, and warfare, often before institutions have agreed on how to govern them.
Legal systems are being forced to adapt. The Ikner case raises questions about what happens when ChatGPT-generated material enters an evidence file: whether it should be treated as testimony, a lead, a record of human intent, or something else entirely. In another incident, a Flock employee called police on a reporter filming a surveillance camera being installed in public—a small but telling example of surveillance systems becoming objects of public scrutiny themselves.
The politics of AI will ultimately be inseparable from the politics of power. A proposed industry-wide standards body could reduce duplication and improve safety, but it could also give a small group of companies disproportionate influence over the definition of acceptable progress. Calls to ban superintelligence outright, including a letter from 40 British MPs to Prime Minister Andy Burnham, express legitimate fears but leave unanswered how such a ban would be enforced, internationally coordinated, or distinguished from ordinary advances in machine learning.
The central question is therefore not simply whether AI should move faster or slower. It is who decides, according to which evidence, and with what ability to challenge the decision. A workable framework will need independent testing, public accountability, international coordination, and room for open research—alongside targeted limits on demonstrably dangerous capabilities. The future of AI governance cannot be left entirely to the companies building the systems, but neither can it be designed without their technical knowledge.
Pacing may be necessary in some areas. It will not be enough on its own. The more durable task is to build institutions capable of distinguishing genuine safety from strategic self-interest—and capable of keeping pace with the technology they are meant to govern.