Search papers, labs, and topics across Lattice.
This paper introduces Cosmos-H-Dreams, a real-time generative physics simulation system for surgical robotics that integrates a generative model with a teacher-student distillation approach. By leveraging a multi-embodiment action-conditioned surgical video world model and achieving approximately 160 inference FPS on a single GPU, the system allows for live control through various interfaces, including surgical robot consoles and VR headsets. The key finding is that Cosmos-H-Dreams enables interactive surgical simulations that can be controlled by both human operators and learned policies, marking a significant advancement in surgical training and decision support systems.
Interactive surgical simulations can now be controlled in real-time by both human operators and AI policies, revolutionizing surgical training and decision-making.
Generative simulation for surgical robotics still lacks real-time interaction. Physical-robot experiments, often involving animal or cadaver labs, are time-consuming, costly, and difficult to reproduce, while classical simulators struggle to capture photorealistic appearance and deformable-tissue dynamics. We address this gap with Cosmos-H-Dreams, an integrated real-time surgical world-model system combining an action-conditioned generative model, a teacher-to-student distillation recipe, and a deployment stack built on the NVIDIA FlashDreams streaming-inference library. Starting from Cosmos-H-Surgical-Simulator, a multi-embodiment action-conditioned surgical video world model fine-tuned on the large-scale Open-H-Embodiment corpus, we post-train this checkpoint on embodiment- and procedure-specific data. By distilling the resulting bidirectional teacher into a causal, few-step student with Self Forcing, we turn a passive video generator into a controllable surgical simulator that streams at $\sim$160 inference FPS on a single NVIDIA RTX PRO 6000 Blackwell workstation GPU. Crucially, Cosmos-H-Dreams is controller-agnostic: any interface that emits a stream of robot kinematics can drive it. We demonstrate live control through a browser keyboard over WebRTC, a Meta Quest headset over WebXR, a commercial surgical robot console such as CMR Surgical's Versius, and learned policies operating in closed loop. To our knowledge, this is the first interactive surgical world model supporting live human and policy control. Human operators and policies alike can act inside the synthesized world and observe the consequences in real time. We release Cosmos-H-Dreams as an open surgical simulation system, providing a common foundation for surgical education, scalable synthetic data generation, and future intraoperative decision support.
