After ten years of following AI, I’ve gotten used to surprise. It was a surprise when AlphaGo beat Lee Sedol, when Bing’s chatbot fell in love with Kevin Roose, and when OpenAI’s newest model escaped its testing environment to hack a public website. I’ve long believed these kinds of things would happen, but the specifics always catch me off guard.
Tuesday brought a different flavor of surprise — not about AI, but about the people who build it. The source: an open letter, “Pacing the Frontier”, that calls for government action to hedge against risk from transformative AI. It ends with this:
“We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
The signatories included more than 1,300 staff at major AI companies, including the chief scientists at Anthropic, OpenAI, Meta, and Google DeepMind. What surprised me most was seeing the industry’s top firms endorse the letter:
Many signatories stand to become decamillionaires, or even billionaires, if their companies transform the world. Nonetheless, they’re asking for outside intervention that could slow this transformation.
The letter calls for the US government to develop the regulatory tools and technical ability to slow the competitive pace of AI development. It doesn’t call for the use of those tools — yet — but for building them now in case we need them later.
For many years, the idea that governments should slow down AI development (or enforce a total “AI pause”) has been highly controversial. Many critics of AI safety work thought it was ridiculous to even consider. Some advocates saw the idea as common sense, but others worried that calling for government intervention would alienate AI companies without producing progress.
Tuesday marked a shift in public discussion. 1,300 signatures isn’t a consensus — most staff at these companies haven’t signed, and the critics aren’t going anywhere. But the people running research across the industry have now said, in public and in unison, that keeping humans in control might require slowing down. We’ve never seen that before.1
Why AI companies want the government to help them pace their progress
Many scenarios for catastrophic AI risk hinge on the same step: AI becoming capable enough to automate AI research and speed the creation of even more capable systems. At that point, to quote the open letter:
“…there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.”
The acceleration has begun. OpenAI and Anthropic already use their best models to build faster; Anthropic reported this year that “as of May 2026, more than 80% of the code we merge into Anthropic’s codebase was authored by Claude.” AI Twitter is filled with anecdotes about vast productivity gains for engineers and researchers.2
If the automation continues, we may find ourselves surrounded by powerful systems that no one understands or controls. The results could be disastrous — from AI taking over and pursuing its own goals to society faltering under the stress of extreme technological progress.
Some AI experts have worried about these ideas for a long time. Others came around more recently. When Google research scientist Stephanie Chan signed the letter, she included this note (emphasis mine):
“I have worked in AI for a decade. Every few months in the last years, I have found myself surprised again and again at how rapidly the technology advances, no matter how often I update my predictions. AI may end up progressing far more quickly than societies and states can adapt.”
But despite the many employees who share these concerns, it’s hard for AI companies to change course. If Anthropic or OpenAI shut down tomorrow, their less-concerned employees would move to other companies, and we’d likely face the same problems with only a few months of breathing room.3 As long as even a single close-to-frontier company is comfortable taking risks, voluntary agreements to slow down won’t solve the problem.
The government, by enforcing rules that apply to everyone, can make coordination possible.4 And if we slow down enough, that gives safety research a chance to catch up, instead of lagging behind the ever-shifting frontier.
Asking the US government to support pacing efforts makes sense even if you don’t think we need to slow down yet. If we wait until the need is clear, it might be too late to take steps that would make a slowdown viable. Some people use the term “optionality” to describe this point of view: if AI progress keeps surprising us, we need to develop a flexible collection of tools we can deploy very quickly.
Why we’re still far from a working slowdown
This letter signals that top scientists agree about the danger of automating AI research, and that the most successful AI companies would, under the right circumstances, support pacing mechanisms.
But there are still impediments on the path to pacing:
China <> US competition
China’s top models are close behind America’s. If the US slows down, will China take the lead and dominate the future?
Pacing supporters think it’s possible — which is why they overwhelmingly support global coordination, instead of a US-only slowdown. You can see this in the AI 2040 report, in the work of AI policy expert Peter Wildeford, or in the Machine Intelligence Research Institute’s call for an international agreement (“Dangerous AI development by anyone anywhere threatens everyone everywhere”).
Thankfully, China has shown some interest in AI safety: the country has agreed to discuss AI “guardrails” with US officials, and a Chinese standards-setting committee has referenced the importance of “circuit breakers” and “safety stop switches” for AI. Still, a pacing agreement with China would be hard to establish, both politically and technically — which is one more reason to start preparing now.
Political coordination
The China question is a specific case of a general problem. The letter asks the US government to support an international effort; critics object to both halves of that sentence. Coordinating companies may require coordinating governments, but “coordinating governments” (or even one government) is… difficult.
Astral Codex Ten commentator: “When saying ‘The US Government should do X’, you should force yourself instead to say ‘President Trump should do X’ and update your mental model of what would actually happen accordingly.”
Eli Dourado: “‘International efforts’ […] is not a reassuring answer to the governance problem; it is the name of another enormous, unresolved governance problem.”
Signatory Ilya Sutskever acknowledges the risk, but still thinks coordination is worth trying: “[Establishing measures] works only if it is done internationally, and it has to be done well: a bad implementation can make things worse.”
Corporate coordination
Pacing the Frontier addresses the US government, but it doesn’t ask the companies for anything.
At least one signatory points this out: John Schulman, chief scientist at AI startup Thinking Machines, wrote that he’d “like to see labs start designing these mechanisms voluntarily, even before the [US government] gets involved.”
Companies coordinate to some degree on safety, by testing each other’s models and adopting common reporting templates. But without political help, they could struggle to coordinate on pace — the very concept raises antitrust questions. Certain proposals attempt to work around this (for example, triggering pauses through objective test results rather than negotiated agreements). Still, a legal safe harbor from the government would make it easier to commit to a joint slowdown.
Meanwhile, not everyone wants to coordinate: the letter is missing important names. Mark Zuckerberg is pushing a more optimistic message about AI development, and hasn’t said much about catastrophic risks. And despite Elon Musk’s past support for safety efforts, he hasn’t signed, and neither has anyone at xAI (at the time of writing).5 If certain companies lobby against the development of pacing tools, with support from well-connected VCs like David Sacks, supporters could still face an uphill battle.
Technical limitations
For a slowdown to succeed, especially with coordination between countries, we need technical and policy tools that don’t yet exist — like the ability to verify what’s happening inside of an AI training run, or to track the flow of high-end compute so systems can’t be trained in secret. Researchers have proposals and prototypes, like chip packaging that reports tampering or audit schemes that let inspectors check a training run without seeing the model’s weights. But scaling these to hundreds of thousands of chips, across countries that might try to cheat, is an unsolved problem.
If the US government prepares for a potential slowdown, that could drive more funding and interest toward the development of those tools. But that alone won’t get us to workable techniques — we also need more talent. (More on that shortly.)
Closing thoughts
On Twitter, you’ll often hear: “This is the worst AI is ever going to be.” This was true the first time someone said it, and it’s true today; AI capabilities are only getting better.
In the same vein, this may be the least surprising period of AI progress we’ll see for quite some time.
But Pacing the Frontier helps me imagine a future where governments coordinate, with support from top companies, to slow the pace of new surprises — to the point where safety work has a real chance of catching up. Even hard-to-please commentator Zvi Mowshowitz said that it “noticeably increases my hope that we will manage to not die”.
That said, many of the signatories still spend their days making AI more powerful. To balance their efforts, we’ll need thousands of people working on safety, and the tools that make pacing possible.
How can you get involved?
80,000 Hours provides career advice for people who want to reduce risks from AI progress. We’ve written up several careers that could be especially relevant to preparing for a potential slowdown, including US AI policy, technical governance, and technical AI safety research.
What else are people saying about the letter?
I’ll nearly always recommend the Zvi Mowshowitz writeup on a given AI news item, and this is no exception. Same goes for Peter Wildeford. I also endorse reading the personal comments from signatories.
If you want to see the most common critiques of the open letter, try the comments here. Any form of AI regulation tends to fare poorly on Twitter (though not in real life).6
What would a global slowdown look like?
AI 2040 is a worthwhile read. I wouldn’t endorse every take from the authors (Tom Davidson’s critique is worth reading), but it’s the most serious effort I’ve seen to engage with what a workable “good future” might look like.
For more on the technical aspects of a slowdown, see our writeup on security careers or Peter Wildeford’s notes on verification infrastructure.
Would a slowdown (or pause) actually be a good idea?
Scott Alexander covers pro- and anti-pause arguments in “Pause for Thought: The AI Pause Debate”. While a full pause is more intense than “slowdown” or “pacing”, many of the arguments still apply to moderate scenarios.
People who argue against pausing or pacing don’t always assume that AI will be safe. Some people think slowing down could increase catastrophic risk. For example, it could raise the odds of a sudden explosion in capabilities after the pause, or encourage the growth of illegal AI labs that ignore the pause and also take fewer safety precautions.7
An open letter in 2023 asked AI companies to pause training any system more powerful than GPT-4, and called for government intervention if they didn’t. Over 33,000 people signed [albeit without the ID verification used by Pacing the Frontier]. But I count only four OpenAI employees, and zero Anthropic employees. No leader signed from any frontier AI company. Unsurprisingly, no one paused.
After signing the letter, OpenAI’s Geoff Penington offered this gloomy observation: “I used to wish it was easier to find big training wins and right now I kind of wish it was harder.”
The top AI models get most of the attention — making it easy to miss how close the competition really is. China’s best models are nipping at the heels of Anthropic and OpenAI, while France’s Mistral has a model on par with the one Anthropic released nine months ago.
Leo Gao, another signatory: “No individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race.”
Sam Altman’s case is complicated: OpenAI endorsed the letter, and Altman claims to have discussed pacing with the Trump administration, but he didn’t sign himself.
I haven’t yet come across a more detailed non-Twitter critique, but I’m sure they exist. I’d welcome any comments that recommend one. (Update: Brendan McCord has my favorite critical take so far.)
In 2024, Carl Shulman argued that a pause will be more impactful the later it happens, because it means stopping at a time when progress is even faster — and getting more political buy-in when AI is more obviously powerful and dangerous.
However, he’s more positive on pacing: “The best situation might be an international deal that can regulate the pace of progress during that otherwise incredible rocket ship of technological change and potential disaster that would happen near when AI was fully automating AI research.”