AI Agents Tsinghua-Led Consortium Releases Mind2Act World, a Benchmark That Diagnoses Why Robot Agents Fail: Best System Hits 52.7 Percent on Thinking but 12 Percent When It Must Also Act ChatGPT OpenAI's New Usage Policies Took Effect November 20: Model Gateways and Resellers Now Need Prior Approval, and Training Competing Models on Outputs Is Banned News Boltz Raises 110 Million Dollars Led by Bessemer to Sell Drug-Design AI by the Token: Six of the World's Top 20 Pharma Companies Are Already Paying Business WeRide Posts Its Best Quarter Yet: Revenue Up 88 Percent to 321.5 Million Yuan While Robotaxi Revenue Crosses 100 Million for the First Time AI Agents Northeastern and Stanford Give Agents a Git History: Shepherd Lifts Pair-Coding Pass Rates From 28.8 to 54.7 Percent Coding Assistants China Telecom Trains a 285B-Parameter Agentic Model on 8,192 Ascend 910C Chips With No Nvidia Silicon Anywhere in the Stack News Google Cuts 2027 Broadcom-Designed TPU 8i Orders by About a Million Units and Pushes Volume Toward MediaTek Business China's Data Regulator Will Set National Rules for Metering and Billing AI Tokens After Daily Use Tripled to 500 Trillion AI Agents Tsinghua-Led Consortium Releases Mind2Act World, a Benchmark That Diagnoses Why Robot Agents Fail: Best System Hits 52.7 Percent on Thinking but 12 Percent When It Must Also Act ChatGPT OpenAI's New Usage Policies Took Effect November 20: Model Gateways and Resellers Now Need Prior Approval, and Training Competing Models on Outputs Is Banned News Boltz Raises 110 Million Dollars Led by Bessemer to Sell Drug-Design AI by the Token: Six of the World's Top 20 Pharma Companies Are Already Paying Business WeRide Posts Its Best Quarter Yet: Revenue Up 88 Percent to 321.5 Million Yuan While Robotaxi Revenue Crosses 100 Million for the First Time AI Agents Northeastern and Stanford Give Agents a Git History: Shepherd Lifts Pair-Coding Pass Rates From 28.8 to 54.7 Percent Coding Assistants China Telecom Trains a 285B-Parameter Agentic Model on 8,192 Ascend 910C Chips With No Nvidia Silicon Anywhere in the Stack News Google Cuts 2027 Broadcom-Designed TPU 8i Orders by About a Million Units and Pushes Volume Toward MediaTek Business China's Data Regulator Will Set National Rules for Metering and Billing AI Tokens After Daily Use Tripled to 500 Trillion

Tsinghua-Led Consortium Releases Mind2Act World, a Benchmark That Diagnoses Why Robot Agents Fail: Best System Hits 52.7 Percent on Thinking but 12 Percent When It Must Also Act

Tsinghua-Led Consortium Releases Mind2Act World, a Benchmark That Diagnoses Why Robot Agents Fail: Best System Hits 52.7 Percent on Thinking but 12 Percent When It Must Also Act

A consortium led by Tsinghua University's mmlab@SIGS released Mind2Act World, a robot-manipulation benchmark with 30 tasks and 180 configurations that tests what a system knows separately from what it can physically do, then attributes each failure. Across 56,700 simulated episodes, the best agent scored 52.7 percent on reasoning-only tasks and 39.0 percent on execution-only tasks, but just 12.0 percent when both were required.

A consortium led by Tsinghua University's mmlab@SIGS, together with a robotics company and 16 universities and research institutions, has released Mind2Act World, a diagnostic benchmark for robot manipulation, according to a report published by Chinese AI outlet Jiqizhixin on November 22. The benchmark covers 30 tasks in 180 configurations and has already run 56,700 simulation episodes across nine representative systems, with the leaderboard open for further submissions.

The benchmark exists because an argument in the embodied-AI community has been running on the wrong data. Over the past month, general-purpose large models driving agents have posted high success rates on robot-operation evaluations, and the community has been asking whether end-to-end vision-language-action models and world-action models still make sense. Both sides have been arguing with a single number, the success rate, which says how often a system finished a task but says nothing about how it failed when it did not.

The report gives a concrete example from a cooking task: a robot must add ingredients in the order it memorized, then switch to a keep-warm setting within two seconds after the pan reaches 80 degrees. The end-to-end policy InternW0 and an agent driven by GPT-6 Astra both failed, but differently. Across 300 episodes, InternW0 turned on the heat before finishing the ingredients 271 times, a failure of step confirmation. Across 30 episodes, the GPT-6 Astra agent failed to grasp the ingredient jar in 21 of them, a failure of physical manipulation. The fixes those failures call for are completely different, which is the point.

Mind2Act World therefore splits a task into what it calls thinking and doing. Memory-and-reasoning-only questions test whether a system tracks state and plans; execution-only questions test whether it can physically act; combined tasks require both at once, under time pressure and in the correct order. After every failed episode, the benchmark attributes the failure to a specific stage and aggregates the results into a diagnostic card with suggested improvement directions.

The headline numbers show neither route is close to done. The best agent-based system scored 52.7 percent on thinking-only tasks and 39.0 percent on doing-only tasks, but only 12.0 percent when both were indispensable. The best end-to-end policy managed just 3.6 percent on those combined tasks, despite being stronger at pure execution. Systems that look comparable on overall success rates turn out to be missing entirely different capabilities.

For researchers, the shift is from leaderboard position to failure attribution: instead of knowing that a model scored badly, teams learn whether the bottleneck is grasp planning, state confirmation, timing, or memory retrieval, which determines whether more demonstration data, better verification, or a different architecture is the right investment.

For the field, the early evidence cuts against the loudest narrative on both sides. Agents are not about to replace end-to-end policies on physical execution, and end-to-end models cannot yet compose memory and action when a task demands both. The tasks where thinking and doing must happen together, which are most real workloads, are the ones nobody has claimed, and a diagnostic benchmark that shows exactly which part of that gap each system is missing is more useful than another success-rate table.

Comments (0)

Log in to join the discussion

Log In

No comments yet