METR Audit Finds Anthropic's Model Progress Nearly Doubled and May Have Crossed Its Own Warning Threshold in July

The nonprofit evaluator METR has published one of the most detailed external audits ever conducted inside a frontier AI lab, and its central finding is striking: Anthropic's rate of model improvement has roughly doubled, and about 80 percent of that acceleration appears to trace back to AI being used in the company's own research. METR's team of 2.5 full-time staffers spent from August 12 to October 29, 2026 inside Anthropic's data, with access to benchmark results from unreleased models, training metrics, employee surveys and internal analyses, plus roughly 15 interviews with researchers. METR took no money from Anthropic and worked on Anthropic-issued devices for security reasons.

The numbers behind the headline: measured on the AECI, a composite metric that bundles many benchmarks, Anthropic's models gained about 15 points per year before October 2025. Since then, the rate has hit 29 — a 1.9x acceleration that packs what used to be two years of progress into one, roughly comparable to the jump from Claude 2.1 to Sonnet 4.5. The analysis included three unreleased models that finished training between July and October 2026. METR says it is confident the speedup is not driven by faster growth in training compute.

Instead, the auditors attribute about 80 percent of the acceleration to AI assisting Anthropic's own research. One employee told the interviewers: "Now only bottlenecked on ideas and compute, no longer bottlenecked on code." Others said experiments that used to take one or two days to set up are now "about free." AI is not yet doing fully autonomous research, but METR found "signs of life" that independent contributions could play a larger role soon, and no bottlenecks that would slow the pace over the next twelve months — meaning there is significant risk the acceleration keeps growing.

The self-reported numbers come with heavy caveats. In informal Slack polls from October 2026, employees said the latest internal model made them roughly 12 times more productive, up from 1.2 times with Claude 3.7 Sonnet. METR treats these self-assessments as "very loose evidence," unrepresentative and likely to overstate actual gains. Internal views differ sharply: one researcher put AI's share of productivity gains at just 15 to 20 percent and distrusted colleagues' self-reports, while another considered new hires the single most important factor of recent months.

METR also stresses the imprecision of its headline figure: the true acceleration factor could be as low as 1.3 or as high as 3. When models from OpenAI, Google and xAI are factored in, an industry-wide estimate drops to just 1.4, possibly because Anthropic lagged competitors in 2025 and a catch-up effect is skewing its numbers.

The audit's credibility problem is Anthropic itself. The company had unilateral redaction rights over all non-public information and used them heavily: according to METR, the majority of the evidence the auditors rated as most informative was stripped out, including forecasts for future models, breakdowns by training steps, and the bottleneck analysis. Anthropic described the redactions as removing confidential information that could accelerate other AI developers. The result, METR writes, is that the public must rely on METR's judgment "much more than would be ideal" — one graph in the report carries the words "FAKE DATA" in large letters to mark redacted figures.

A companion METR review sharpens the point. In its public August 2026 risk report, Anthropic stated the acceleration was "less than a factor of 2." The internal, unredacted version implies a value of 1.97, and Anthropic itself calls it "plausible" that the threshold has been crossed. Under Anthropic's Responsible Scaling Policy, if the pace of AI progress doubles because of automated AI research, a warning level should trigger. METR considers it possible that warning level was already reached on July 15, and accuses the public report of understating both the acceleration and the uncertainty around it. METR was originally supposed to also verify how Anthropic splits compute between capability and safety research; Anthropic canceled that part of the project midway, saying it wanted to focus on risk assessment first. Daniel Kokotajlo, the researcher behind the "AI 2027" scenario, responded sharply on X, arguing that auditing compute allocation is the foundation that makes slowdown commitments verifiable at all.

METR currently shares Anthropic's assessment that acute catastrophic risk is "low," but says it cannot rule out another sharp acceleration over the next six months, which could quickly push the risk rating higher. The episode is a preview of what AI governance actually looks like in practice: a lab invites auditors in, redacts most of what they find, and the public is left to trust the auditor's characterization of evidence it cannot see.