NVIDIA Open-Sources Cosmos3-H-Dreams, a Surgical World Model Trained on Real Operations That Runs Live on One GPU

NVIDIA released a research model on October 20 that it says marks a step change for surgical robotics simulation: Cosmos3-H-Dreams, a generative world model that produces surgical video in response to live instrument commands. Instead of replaying recorded footage, the model starts from a single frame and generates what happens next as a surgeon — or a learned robot policy — moves the tools. The checkpoint and companion code went up the same day on Hugging Face and GitHub under the NVIDIA Open Model License.

What separates this release from its predecessor is the training data. The earlier Cosmos-H-Dreams, published in July, learned from tabletop suturing on the da Vinci Research Kit, a bench task in academic labs. The new model was fine-tuned on clinical cholecystectomy — gallbladder removal — recordings captured on the CMR Surgical Versius platform: 4,792 episodes totaling roughly 17 million frames at 60 Hz and 1080p, subsampled to 10 Hz for training. NVIDIA calls it, to its own knowledge, the first real-time generative surgical simulator trained on clinical recordings that supports live kinematic control.

Under the hood, the model builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter foundation model pretrained on the Open-H-Embodiment dataset, and specializes it into a final model of roughly 15.17 billion parameters. The architecture is a Mixture-of-Transformers that pairs a Qwen3-VL-8B-Instruct reasoning tower with a latent-diffusion generation expert. The action interface carries 28 dimensions per time step — 14 per arm — covering anchor-relative translation, 6D rotation, jaw position, electrosurgical energy activation and thumbstick inputs, extended from 20 dimensions for the Versius embodiment.

Real-time interaction required a two-stage training pipeline. A bidirectional teacher was first adapted on 24 future frames, then extended to a 72-frame horizon — 2.4 and 7.2 seconds at 10 Hz. Distillation then converted the teacher, which needed 35 sampling steps, into a causal student via teacher forcing and Self-Forcing with distribution matching. Two student checkpoints are released: a four-step version favoring image quality and a separately trained two-step version trading visual fidelity for speed. Streaming runs through FlashDreams, the accelerated inference library, with an adapted FadeMem module that consolidates the growing key-value cache during long rollouts. The whole system runs interactively on a single NVIDIA RTX PRO 6000 GPU.

NVIDIA is explicit about the limits. Visual plausibility, the company writes, is a starting point for validation, not proof of biomechanical accuracy — a convincing-looking scene does not establish that tissue deforms the way real tissue would. The system has not been validated for patient care, is not cleared for clinical decision-making, and connected controllers must match the action conventions used in training. The stated intended uses are surgical robotics policy evaluation in simulation, interactive teleoperated world-model exploration, and synthetic data generation for robotic cholecystectomy research. All capability claims in this article are company-published and have not been independently verified.

The release extends the Medical Physics Simulation framework NVIDIA open-sourced on July 22 within its Isaac for Healthcare stack, where conventional solvers handle contact and friction while a learned model supplies visual scene dynamics. CMR Surgical, which contributed nearly 500 hours of anonymized Versius procedure data to Open-H-Embodiment and describes itself as the largest contributor to that dataset, demonstrated a live controller hookup at a robotic surgery conference in Florida earlier this year.

The bottleneck this model targets is structural: real operating-room data is scarce, expensive and ethically fraught, which caps how many environments a surgical robot policy can be trained and scored against. A generative simulator trained on genuine clinical footage offers, in principle, unlimited practice rooms that behave like the real thing. Whether it actually does depends on the validation agenda NVIDIA itself lays out — broader procedure coverage, checks of simulated motion against physical measurements, and surgeon-reviewed benchmarks. Until those land, Cosmos3-H-Dreams is best read as a research testbed that makes surgical simulation interactive, not a clinical tool.