Microsoft Says Its Diagnostic Orchestrator Beats Doctors 83 to 36 Percent on Hard Cases, With a Catch

Microsoft published a peer-reviewed study in Science on Thursday claiming that a multi-agent system it calls Diagnostic Orchestrator can diagnose complex medical cases far more accurately than generalist physicians. On several hundred case reports drawn from The New England Journal of Medicine, the system reached about 83 percent diagnostic accuracy. Generalist physicians with access to reference materials scored roughly 36 percent; without references, about 22 percent.

The gap held on a second, independent test set. On emergency department cases taken from a patient admissions database at Beth Israel Deaconess Medical Center in Boston, the system scored 80 percent against an average of 47 percent for the human physicians in the comparison. Microsoft also says its system tended to reach diagnoses at a lower estimated cost, because it orders tests selectively instead of running a broad panel.

The architecture is the interesting part. Rather than asking one model for an answer, Diagnostic Orchestrator, internally abbreviated MAI-DxO, runs a panel of agents that question the patient, decide which tests to order, weigh the results and argue toward a diagnosis, imitating the way a clinical team works a case. Microsoft built its own evaluation harness for this, SDBench, which asks for step-by-step clinical reasoning instead of a multiple-choice answer, and says it will open-source the benchmark.

The paper is candid about where the improvement comes from. Tested with OpenAI models, the orchestration layer added only modest gains on top of strong base models. GPT-5 running alone with a short prompt scored 78.9 percent on the complex cases at roughly 8,400 dollars per case; adding the orchestrator cut estimated cost by about 3,000 dollars but lifted accuracy by only about four points. With GPT-5.5 the split was better, moving accuracy from 72 percent to nearly 87 percent at slightly lower cost. On the emergency department cases, orchestration produced almost no gain at all, with both configurations landing near 85 percent.

The pattern repeats across vendors. Gemini 2.5 Flash scored 46.4 percent alone and nearly 69 percent with the orchestrator, so weaker models benefit far more than frontier ones. The researchers acknowledge that the headroom shrinks as models approach the ceiling, and that the absolute difference between a weak model plus orchestration and a strong model alone is still larger than the orchestration boost itself.

Two limits deserve emphasis. The NEJM cases used in the comparison are the kind normally routed to specialists or multidisciplinary teams, and no specialists took part in the evaluation. The published paper also predates the current frontier generation: GPT-6 Astra, Meta Muse Wonder Full, Google Gemini 4 Argon and Anthropic Claude Opus 5.5 are all absent from the benchmark run.

Mustafa Suleyman, chief executive of Microsoft AI, framed the result as a step toward what he calls medical superintelligence, telling reporters that systems indistinguishable from human performance are a couple of years away, that Microsoft is training a healthcare foundation model with Mayo Clinic for release next year, and that 2027 could bring a coding-style breakthrough for medical records, with streaming predictions about how a case may unfold. Dominic King, vice president of health at Microsoft AI, pointed to the widening mismatch between demand for and supply of medical expertise as the real target.

What the study establishes is narrower than the headlines suggest, and more useful. Multi-agent orchestration is a real but second-order lever: it buys accuracy mostly where the base model is weak, and it buys cost savings by making test ordering more deliberate. The clinical question that remains unanswered is not whether a panel of agents can outscore a lone general practitioner on a written case, but whether such a system improves outcomes when a specialist team is already in the room.