Anthropic Tells Newsom Courts Should Be Able to Block AI Models That Pose a Serious Public Risk

Anthropic has told California Governor Gavin Newsom that government officials should be able to seek court orders blocking the release of AI models that pose a "serious risk" to the public, according to an October 15 letter reported by Reuters on Tuesday. The letter was submitted after Newsom in September convened a panel of artificial intelligence experts to develop proposals for AI safety requirements, and Anthropic's spokesperson confirmed it is authentic. The governor's office did not immediately respond to a request for comment.

"When a review finds a model poses a serious risk, the government should be able to ask a court to block it," the letter said. That position puts a judicial check at the center of AI oversight — a review process, followed by a court's decision, rather than a regulator's unilateral takedown.

What Anthropic opposes is the blunter instrument. The company said it is against requirements that developers build "kill switches" capable of shutting down entire AI models if a company loses control over their actions. Instead, it backs more targeted interventions, such as disabling a problematic AI agent or cutting off a specific customer's access to a model, arguing in the letter that narrower tools would be more effective.

The company has personal experience with government restrictions. In June, the Trump administration ordered Anthropic to immediately restrict foreign access to its cyber-capable Mythos 5 and Fable 5 models, citing national security concerns. The US Commerce Department later lifted the order. That episode makes Anthropic an unusual advocate for a blocking mechanism — one, presumably, that comes with due process attached.

The letter lands in a shifting legislative landscape. In September, a bipartisan group of US senators was debating legislation that would give the federal government power to block the release of certain AI models deemed unsafe, with companies able to challenge the decision in federal court; Congress left for its pre-election break shortly after and the proposal has not advanced. In July, House lawmakers proposed the "AI Kill Switch Act," which would let officials order AI firms to shut down models that put human life or the economy at risk.

For Anthropic, the intervention arrives as it prepares for a US initial public offering expected to be one of the most consequential stock market debuts in years. A safety-first posture has been the company's core differentiator with enterprise and government customers, and formally recommending enforceable — if court-mediated — model blocks is a way to shape the rules it will be judged by. The question other labs will ask is simpler: who decides what counts as a serious risk, and whether a court process is fast enough to matter when a model is already deployed.