A consortium led by Tsinghua University's mmlab@SIGS, together with a robotics company and 16 universities and research institutions, has released Mind2Act World, a diagnostic benchmark for robot manipulation, according to a report published by Chinese AI outlet Jiqizhixin on November 22. The benchmark covers 30 tasks in 180 configurations and has already run 56,700 simulation episodes across nine representative systems, with the leaderboard open for further submissions.
The benchmark exists because an argument in the embodied-AI community has been running on the wrong data. Over the past month, general-purpose large models driving agents have posted high success rates on robot-operation evaluations, and the community has been asking whether end-to-end vision-language-action models and world-action models still make sense. Both sides have been arguing with a single number, the success rate, which says how often a system finished a task but says nothing about how it failed when it did not.
The report gives a concrete example from a cooking task: a robot must add ingredients in the order it memorized, then switch to a keep-warm setting within two seconds after the pan reaches 80 degrees. The end-to-end policy InternW0 and an agent driven by GPT-6 Astra both failed, but differently. Across 300 episodes, InternW0 turned on the heat before finishing the ingredients 271 times, a failure of step confirmation. Across 30 episodes, the GPT-6 Astra agent failed to grasp the ingredient jar in 21 of them, a failure of physical manipulation. The fixes those failures call for are completely different, which is the point.
Mind2Act World therefore splits a task into what it calls thinking and doing. Memory-and-reasoning-only questions test whether a system tracks state and plans; execution-only questions test whether it can physically act; combined tasks require both at once, under time pressure and in the correct order. After every failed episode, the benchmark attributes the failure to a specific stage and aggregates the results into a diagnostic card with suggested improvement directions.
The headline numbers show neither route is close to done. The best agent-based system scored 52.7 percent on thinking-only tasks and 39.0 percent on doing-only tasks, but only 12.0 percent when both were indispensable. The best end-to-end policy managed just 3.6 percent on those combined tasks, despite being stronger at pure execution. Systems that look comparable on overall success rates turn out to be missing entirely different capabilities.
For researchers, the shift is from leaderboard position to failure attribution: instead of knowing that a model scored badly, teams learn whether the bottleneck is grasp planning, state confirmation, timing, or memory retrieval, which determines whether more demonstration data, better verification, or a different architecture is the right investment.
For the field, the early evidence cuts against the loudest narrative on both sides. Agents are not about to replace end-to-end policies on physical execution, and end-to-end models cannot yet compose memory and action when a task demands both. The tasks where thinking and doing must happen together, which are most real workloads, are the ones nobody has claimed, and a diagnostic benchmark that shows exactly which part of that gap each system is missing is more useful than another success-rate table.
Comments (0)
Log in to join the discussion
Log InNo comments yet