Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Business Daniel Ek Is Backing Europa Fusion, a Stealth UK Startup Led by a Former DeepMind Researcher Betting on Pulsed Power Coding Assistants Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement Security A Transcription Firm Used by Police and Prosecutors Left Interview Transcripts and Body-Cam Footage on an Open S3 Bucket News UN Killer-Robot Talks End With a 2028 Deadline and No Binding Rules. About 100 States Wanted a Treaty; the US, Russia and Israel Blocked It Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Business Daniel Ek Is Backing Europa Fusion, a Stealth UK Startup Led by a Former DeepMind Researcher Betting on Pulsed Power Coding Assistants Stripe Tested 14 Coding Agents on 69 Payment Scenarios: 78% of Criteria Passed, but Only 29% of Runs Met Every Requirement Security A Transcription Firm Used by Police and Prosecutors Left Interview Transcripts and Body-Cam Footage on an Open S3 Bucket News UN Killer-Robot Talks End With a 2028 Deadline and No Binding Rules. About 100 States Wanted a Treaty; the US, Russia and Israel Blocked It

METR Updates the Most Misunderstood Graph in AI: Models Still Cannot Design a Good Evaluation

METR Updates the Most Misunderstood Graph in AI: Models Still Cannot Design a Good Evaluation

METR has published its Q4 2026 capability update, extending the famous time-horizon analysis to judgment-heavy, hard-to-verify tasks. Not one AI-designed evaluation was deemed useful by METR staff, though the newest models now score borderline above the bar on that task, where every model a year ago was clearly below it.

METR, the nonprofit that measures what AI models can actually do, has released its Q4 2026 capability update, and with it a revision of the single most cited and most misread chart in the field: the time-horizon plot. The original version showed that the length of tasks frontier models can complete reliably has been doubling roughly every four months. That finding fueled a year of projections about automated research, and METR itself has now updated the work to address what the plot leaves out.

The methodology explains both the power and the blind spot. METR scores task difficulty by how long it takes a human to complete the task, then rates models by the length of tasks they can finish. The original suite was dominated by verifiable tasks, the kind where a script can check success or failure automatically. That is exactly where reinforcement learning works best, which is why models have become superhuman at coding and math: researchers can run a model against a problem thousands of times and reinforce what works. The concern has always been whether those gains transfer to the messy tasks where success is hard to measure, like choosing a research direction, the tasks that matter most if AI is ever to automate research and development itself.

The new report adds tasks built around judgment, and the results are sobering. In one, models were asked to design new AI evaluations that METR could use in its own research, with METR staff scoring the results. Even though the most advanced models now beat nearly every human at software engineering, not one AI-designed evaluation was considered useful for METR's actual work. In a second task, models scored philosophical arguments as a proxy for judgment, and again fell short of human performance. METR itself flags a methodological caveat: a single employee rated most of the arguments, so agents were effectively judged by how well they agreed with one particular person.

The trajectory, however, is not flat. The newest models, such as Opus 5.5, now achieve what METR calls borderline above the bar performance on the evaluation-design task, not yet usable but promising, while a year ago every model scored clearly below the bar. The complication is that METR includes only a handful of these judgment-heavy tasks, because designing and scoring them is expensive and slow. Matt Putz, a member of technical staff at METR, said in an email that by their nature, hard-to-verify tasks are much harder to study and quantify, and that clean, robust trends like the ones for easy-to-specify tasks may simply never exist for this class of work.

Context matters here. Earlier this month a heavily redacted METR report claimed AI may be accelerating the pace of AI progress at Anthropic by a factor of two, stoking fears of recursive self-improvement at the company that had just completed the largest IPO in history. Separately, OpenAI reported this week that its models perform worse on the real-world tasks its own employees assign than on METR's original verifiable suite. Both data points point the same direction: the exponential is real, but it is an exponential on verifiable work.

Experts largely welcomed the expansion. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said AI evaluations need to get more creative and called pushing the envelope of the evaluation space a welcome change. He was not surprised by the shortfall, pointing to an August preprint in which he and collaborators found models cannot yet reproduce human computer science research on their own. Najoung Kim, a linguist and computer scientist at Boston University, praised the move but cautioned about extrapolation: the observed improvement tells us something about the current moment, but she is not sure yet how to project it forward.

The takeaway for anyone tracking claims of imminent automated AI research: the honest picture is two curves, not one. On verifiable tasks, progress is steep and showing no sign of slowing. On judgment-heavy work, the kind that decides which problems are worth solving, the best models are still below the humans who would be replaced, even as they close the gap from below. Research taste remains the bottleneck, and nobody currently knows how to measure it, let alone train it.

Comments (0)

Log in to join the discussion

Log In

No comments yet