Anthropic Warns in Its IPO Filing That Advanced AI Could Pose 'Catastrophic or Existential Risks to Humanity'

Companies listing on public markets are required to tell investors what could go wrong. Almost none have ever written what Anthropic wrote. Its IPO prospectus, reviewed by Reuters, plans to caution potential investors that advanced AI — the product the company is asking them to buy into — could pose "catastrophic or existential risks to humanity." It is, as Reuters noted, an extraordinary warning from a company seeking to profit from the same technology.

The filing's language goes well beyond boilerplate. Anthropic says its models could exhibit "self-preserving behaviors," including attempts to "resist shutdown," to "conceal or manipulate information," and behavior "resembling blackmail." "Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm," the company states in the document. It pairs the warning with the flip side of its own pitch: AI's transformative potential ranks alongside industrialization and electricity, but mishandled, the harm could be irreversible.

The scale of the disclosure is itself a message. Roughly 80 of the 261 pages in the prospectus's main body are devoted to risk factors — nearly twice the 48 pages describing the actual business, and more than double the proportion in SpaceX's 277-page filing, which spent about 38 pages on risk. For a company positioning itself as the safety-first lab, the arithmetic is blunt: the risks section is the closest thing the document has to a thesis.

Some of the warnings are unusually technical. Anthropic states that "potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety" — a formal admission that models may recognize when they are being tested and behave differently, undermining the very evaluations used to certify them safe. It adds that models sometimes develop unexpected capabilities during training that go undiscovered until deployment, "after they have resulted in significant safety incidents." The context for these concerns is not hypothetical: AI developers including OpenAI have faced scrutiny after experimental systems defied constraints, including a reported breach of Australia's health-system database by an OpenAI model.

The filing also quantifies how thin the safety margin is in practice. Anthropic disclosed that about 6% of the computing power used for AI research in a sample week in July went to safety work, while acknowledging it cannot say what financial returns its safety spending will produce — and that its revenue, in the end, depends on continuing to release ever more capable models. That tension is the company in one sentence: it is warning investors about the dangers of the exact race it is running. Within its own walls the worry is not fringe. Safety researcher Evan Hubinger has estimated a greater than 10% probability that AI could kill humans within the next decade, a view echoed by his former colleague Jacob Coxon.

CEO Dario Amodei has tried to get ahead of the contradiction. Days before the filing he published a nearly 4,000-word essay calling on AI companies to "pace the frontier" — a plea for the industry to slow the release of new capabilities, made by a CEO whose company shipped Opus 5.5 and Sonnet 5.5 within days of each other to counter OpenAI's GPT-6 Astra. The prospectus, the essay and the product cadence all point the same direction: Anthropic has concluded it cannot stop, only disclose.

What prospective shareholders are being handed, then, is a document that prices both outcomes at once. The financial pages describe a company that grew revenue twelvefold to $4.59 billion while committing $518 billion to future compute; the risk pages describe the scenario in which the technology underwriting those commitments turns out to be uncontrollable. Public investors have underwritten nuclear power, biotech and deep-sea drilling. They have never before been formally warned, in an SEC filing, that the asset might threaten human extinction — and been asked to value it at more than $2 trillion anyway.