Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days

29 Institutions Just Built a Report Card for Medical AI, and Retested Every Score Themselves

29 Institutions Just Built a Report Card for Medical AI, and Retested Every Score Themselves

InclusionMed, billed as the first independent evaluation and analysis platform for healthcare AI, launched with an atlas crossing patient, professional and administrator roles against knowledge, reasoning and task execution. Its first index covers 5 benchmarks, 1034 tasks and 22 models from 11 vendors, all independently retested rather than accepted from vendor submissions. One caveat: Ant Group is both a co-builder and the source of the launch coverage.

Medical AI has a benchmark problem: nearly every narrow task has its own leaderboard, yet almost none of them answer the question a hospital actually asks — does this model hold up in real clinical work? A new platform called InclusionMed launched this week to answer that, and it is doing so with an unusual amount of institutional weight behind it: researchers from 29 organizations including Stanford University, Harvard University, Peking Union Medical College, Ant Group and InclusionMed.AI. The project is being described as the healthcare counterpart to Artificial Analysis, the independent AI model index.

The structure is the interesting part. InclusionMed Atlas organizes every benchmark along two dimensions: who the tool serves — patients and the public, medical professionals, healthcare administrators — and what capability it tests — knowledge, reasoning, or task execution. Crossing the two produces a nine-cell grid, and each benchmark is filed into the cell it primarily measures. That mapping makes the gaps visible instead of implicit: you can see at a glance that clinical knowledge is saturated with leaderboards while administrative scheduling is nearly empty.

The first composite index draws on five benchmarks covering 1,034 tasks to compare 22 models from 11 vendors, domestic and international, focused on two areas: clinical diagnosis and treatment, and healthcare management. The organizers say every score was independently retested by the platform rather than taken from vendor submissions, and that benchmarks were selected by weighing medical relevance, evaluation credibility and complementarity across scenarios. The inaugural index combines four public benchmarks with the team's own InclusionMed-DxFiles, a set of difficult cases built from top medical journals that tests whether a model can combine scattered case clues into a clinical judgment — closer to how physicians actually reason than to textbook recitation.

Why does an independent grader matter here? Because hospitals buying AI have had no neutral way to compare models, and vendors have every incentive to publish selectively — the benchmark they top, on the task they chose, with the scoring they wrote. A multi-party index spanning roles and capability types, with retesting, at least gives procurement a shared vocabulary and puts pressure on cherry-picked claims. It also gives clinicians a way to say where a model's competence ends, which matters far more in a clinical setting than a general chatbot score.

The caveats are real, though. Ant Group is simultaneously a co-builder of the platform and the organization that supplied the launch article to QbitAI, so the independence claim is, for now, the organizers' own description of themselves. The launch coverage does not name the four public benchmarks, the vendors or the 22 models, and publishes no per-model scores — so there is nothing yet to audit. Whether the first index holds up depends entirely on that disclosure: names, scores and methodology, released fast enough to be checked.

The team says it is already working with medical institutions on new evaluation tasks, including shared doctor-patient decision-making in complex treatment choices and care scheduling under resource constraints. That direction — evaluating models inside the actual workflow constraints of hospitals, not just on exam questions — is the right one, because the difference between a model that answers well and a model that can be deployed is where most medical AI projects quietly die.

The larger point is about power. Whoever writes the exam controls what gets measured, and what gets measured decides what gets built. If InclusionMed can keep its grading independent of the companies on its scoreboard, it becomes infrastructure; if not, it is just another leaderboard. The industry has plenty of the latter.

Comments (0)

Log in to join the discussion

Log In

No comments yet