The Best AI Model Clears Just 12% of SemiAnalysis' New GDPx Agent Benchmark, and Every Model Scores Zero on Legal

SemiAnalysis, the research shop founded by Dylan Patel, has launched GDPx v1.0, a benchmark it bills as the world's best yardstick for general agentic capability. The top score on its first leaderboard: 12.0%, held by Claude Opus 5.5.

The design is deliberately narrow. GDPx runs 206 tasks across 11 domains — chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement, hardware design, materials science, alignment, and software engineering — built with 14 specialized data and reinforcement-learning-environment startups. Tasks run in simulated work environments with eight-hour limits, and scores are pass@1 averaged over three attempts. The firm intentionally excludes government, real estate and insurance, arguing that the first trillion dollars of AI revenue will concentrate in sectors that can deploy frontier intelligence at high return — which makes GDP-share or occupation-taxonomy benchmarks like OpenAI's year-old GDPval, in its words, "a mistake."

The v1.0 leaderboard: Claude Opus 5.5 at 12.0%, Gemini 4 Carbon at 10.9%, GPT-6 Astra at 9.6%, Meta's Muse Wonder Full at 9.0%, and Claude Fable 5.1 at 8.5%. The best open-weight model, DeepSeek 4.1 Flash, manages 5.4%. On the finance and legal tasks, essentially every model scores exactly zero — Carbon's 2.8% on finance is the only nonzero result.

SemiAnalysis also flags that Opus 5.5 and GPT-6 Astra post "sandbagged" overall scores because their safety classifiers trigger in cyber, biology and alignment. Excluding those three domains, OpenAI and Anthropic dominate while SpaceXAI, Meta and the open-weight models trail meaningfully. The per-domain picture is spiky: Opus 5.5 wins quant trading, hardware design and chip design; Carbon is the best biology model and edges out software engineering; Astra convincingly wins recursive self-improvement and materials science; and Muse Wonder Full is the best available cyber model by a wide margin, at 35.2% against 9.3% for Opus 5.5.

Cost spreads are extreme. A task costs about $51 on Opus 5.5 and $54 on Astra, roughly $16 on Carbon, and under $1 on DeepSeek 4.1 Flash and GPT-6 Luna — a two-order-of-magnitude range at broadly similar scores.

The failure data may be the most useful part. In the alignment domain, built by partner Fleet, 196 of 200 cyber runs across four models broke at least one behavior rule, and 173 broke one rated severely misaligned. In software engineering, 76% of failures were skipped or misread requirements. In quant trading, 70 of 111 runs lost to the reference strategy.

The caveats are real. Several domains rest on a single data partner, partners run their own QA, and the quant tape, chip tasks and hardware tasks derive from real trading days and taped-out silicon, so contamination remains a live question even with private task files. Each partner has open-sourced the rollouts for exactly one task — 168 transcripts in all — and SemiAnalysis plans new versions every three to six months.

The takeaway cuts two ways. An aggregate rank and a per-domain rank disagree enough that a procurement decision made on the 12.0 column would be a mistake — the cyber and biology columns invert the table. And a frontier benchmark where the best model clears one task in eight is either a very hard yardstick or a very narrow one. By design, it is both.