Artificial Analysis, the independent benchmarking shop best known for its model intelligence index, launched AA-Robotics v0.1 on October 14 with a simple framing: "welcome to the era of embodied AI." The research preview measures how well frontier language models can understand, plan and act in the physical world, and its first scoreboard is as much an indictment of cost and speed as it is a showcase of capability.
The setup is deliberately unglamorous. Each model controls a real six-axis arm, the Hiwonder NexArm, which has five joints plus a gripper and is driven over an HTTP API. Four cameras watch the scene, three mounted around the table and one on the wrist. The model runs inside a coding-agent harness, receives only brief instructions covering how to connect, which arm it is driving and how to submit a task, and then has to write and execute its own motion-planning code in an isolated environment. Eight tabletop tasks, including grasping, placing objects and pressing buttons, are scored out of 100, with partial credit for incomplete runs.
GPT-6 Astra leads at 69, followed by GPT-6.1 Sol at 65 and Gemini 4 Argon at 55, with Claude Opus 5.5 and Claude Fable 5.1 landing in the 46-49 range. The economics are the story. A task takes about 30 minutes on average, roughly 100 times slower than the human baseline, and about 10 times more expensive. Sol completes a task for about $1.93; Astra needs $16.36. Astra's four-point lead over Sol costs $14.43 more per task, more than eight times Sol's price. The GPT-6 models reportedly also took 75 to 80 seconds before their first movement.
The anecdotes are where the reality gap shows. One Sol-driven arm spent more than ten minutes stacking foam blocks into an acceptable tower, then, having completed the task, kept adjusting it until the run was scored a failure. On a separate set of 11 humanoid tasks that require the model to keep a robot balanced and walking on its own, coverage of the launch reports that no model reached 2 out of 100, and that Grok 4.7 and Gemini 4 Argon failed to move the robot at all in most episodes.
The caveats matter as much as the numbers. Version 0.1 is a research preview: the launch post and headline scores are public, but the full methodology, task list and raw episode logs were not published at launch, and one independent review noted that the dedicated robotics page returned a 404. How episodes are scored, how many runs per task, and exactly what hardware configuration was used have therefore not been independently replicated, and every figure above should be read as the benchmark owner's own result rather than a settled measurement.
The benchmark lands in the middle of an argument the industry has been having all autumn. Last month GPT-6 Astra drew the Golden Gate Bridge with a roughly $150 arm and put 19 of 20 blocks into a bowl using two arms in RoboCurve's real-robot tests, videos that prompted an uncomfortable question for robotics companies: if general models already see, reason and can write control code, why train a bespoke vision-language-action model at all? Founders of Chinese humanoid startups have pushed back, arguing at September's Bund Summit that the displacement thesis does not hold yet, because model capability follows data: coding models are trained on mountains of verified code, while robotics still lacks comparable data, evaluation infrastructure and hardware.
The most useful number in this release is not on the leaderboard. Astra's win at sixteen times Sol's price shows capability outrunning economics, and a 100x speed gap is the kind of figure that tends to collapse quickly once a problem becomes a benchmark target. The number to watch in v0.2 is not points; it is minutes per task.
Comments (0)
Log in to join the discussion
Log InNo comments yet