Alibaba-Led Benchmark Finds the Best Coding Agent Fully Finishes Only 26.4 Percent of Real Requests, Against 99-Plus Percent on Saturated Tests

Agentic coding models score above 99 percent on the benchmarks most companies use to pick them. A new evaluation built from real user requests suggests those numbers measure a world that no longer exists: on tasks that actual users struggled to get done, the best model fully finishes barely one in four.

Co-Design Bench, described in a paper posted to arXiv on October 26 by researchers from Alibaba Group, Fudan University, Zhejiang University and other Asian institutions, is constructed from 37,463 deduplicated change requests drawn from 26,314 real coding sessions. The authors deliberately favored cases where users had to rephrase requests, send repeated follow-ups or roll back changes — in other words, tasks that already gave agents trouble. Experts refined each request and wrote 2,331 hidden grading criteria across 273 tasks, spanning localized multimodal edits (where agents may receive images, audio, video or documents) and repository edits that require coordinated changes across a codebase.

Grading is deliberately hostile to shortcuts. One AI judge tests the finished app in a browser; a second re-checks every positive verdict and hunts for mock-ups or hardcoded results. Points go only to features that actually work. Under that regime, OpenAI's GPT-6 Astra, the strongest of ten model endpoints tested, met 69.9 percent of acceptance criteria per task on average but fully satisfied every criterion on just 26.4 percent of tasks — 41.0 percent on multimodal edits but only 17.9 percent on repository work. Claude Opus 5 followed at 66.1 and 18.3 percent, Alibaba's Qwen3.8-Max at 65.3 and 16.5 percent, and xAI's Grok 4.6 landed last at 51.1 and 6.6 percent.

Two structural findings stand out. First, every model degraded when a task combined three or four kinds of changes — layout, design, interaction and content — which the authors note is the norm in real projects, not the exception. More compute does not rescue this: GLM-5.2 burned a median of about 7.29 million tokens per multimodal task against GPT-6 Astra's roughly 1.24 million and still scored meaningfully worse. Second, the harness matters as much as the model. Running Qwen3.8-27B through different agent environments moved its repository score from 0.560 to 0.749, and reshuffled which of two models led depending on the tool. The authors' recommendation: leaderboards should report model and harness together, and buyers should test the pair.

The benchmark has stated limits. Each request is fixed — no follow-up questions or mid-task corrections — and AI judges can err. But its findings rhyme with earlier independent work: a METR study found the real acceptance rate of AI-generated code by project leads ran roughly 24 percentage points below scores on SWE-bench Verified, and a separate analysis of coding-agent harnesses found elaborate setups consumed up to three times the tokens with barely measurable benefit.

For enterprises budgeting for agentic coding tools, the practical read is that a 26.4 percent end-to-end completion rate for the frontier model, against a 69.9 percent average requirement coverage, means most outputs still leave at least one unfinished detail — and human review remains a cost line rather than a transitional one, at least until benchmarks stop flattering the products they evaluate.