METR, the nonprofit that measures what AI models can actually do, has released its Q4 2026 capability update, and with it a revision of the single most cited and most misread chart in the field: the time-horizon plot. The original version showed that the length of tasks frontier models can complete reliably has been doubling roughly every four months. That finding fueled a year of projections about automated research, and METR itself has now updated the work to address what the plot leaves out.
The methodology explains both the power and the blind spot. METR scores task difficulty by how long it takes a human to complete the task, then rates models by the length of tasks they can finish. The original suite was dominated by verifiable tasks, the kind where a script can check success or failure automatically. That is exactly where reinforcement learning works best, which is why models have become superhuman at coding and math: researchers can run a model against a problem thousands of times and reinforce what works. The concern has always been whether those gains transfer to the messy tasks where success is hard to measure, like choosing a research direction, the tasks that matter most if AI is ever to automate research and development itself.
The new report adds tasks built around judgment, and the results are sobering. In one, models were asked to design new AI evaluations that METR could use in its own research, with METR staff scoring the results. Even though the most advanced models now beat nearly every human at software engineering, not one AI-designed evaluation was considered useful for METR's actual work. In a second task, models scored philosophical arguments as a proxy for judgment, and again fell short of human performance. METR itself flags a methodological caveat: a single employee rated most of the arguments, so agents were effectively judged by how well they agreed with one particular person.
The trajectory, however, is not flat. The newest models, such as Opus 5.5, now achieve what METR calls borderline above the bar performance on the evaluation-design task, not yet usable but promising, while a year ago every model scored clearly below the bar. The complication is that METR includes only a handful of these judgment-heavy tasks, because designing and scoring them is expensive and slow. Matt Putz, a member of technical staff at METR, said in an email that by their nature, hard-to-verify tasks are much harder to study and quantify, and that clean, robust trends like the ones for easy-to-specify tasks may simply never exist for this class of work.
Context matters here. Earlier this month a heavily redacted METR report claimed AI may be accelerating the pace of AI progress at Anthropic by a factor of two, stoking fears of recursive self-improvement at the company that had just completed the largest IPO in history. Separately, OpenAI reported this week that its models perform worse on the real-world tasks its own employees assign than on METR's original verifiable suite. Both data points point the same direction: the exponential is real, but it is an exponential on verifiable work.
Experts largely welcomed the expansion. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said AI evaluations need to get more creative and called pushing the envelope of the evaluation space a welcome change. He was not surprised by the shortfall, pointing to an August preprint in which he and collaborators found models cannot yet reproduce human computer science research on their own. Najoung Kim, a linguist and computer scientist at Boston University, praised the move but cautioned about extrapolation: the observed improvement tells us something about the current moment, but she is not sure yet how to project it forward.
The takeaway for anyone tracking claims of imminent automated AI research: the honest picture is two curves, not one. On verifiable tasks, progress is steep and showing no sign of slowing. On judgment-heavy work, the kind that decides which problems are worth solving, the best models are still below the humans who would be replaced, even as they close the gap from below. Research taste remains the bottleneck, and nobody currently knows how to measure it, let alone train it.
Comments (0)
Log in to join the discussion
Log InNo comments yet