Search papers, labs, and topics across Lattice.
The paper introduces Spatial-ORMLLM, a novel multimodal large language model designed for 3D spatial reasoning in operating rooms using only RGB images. It addresses the limitations of existing methods that rely on extensive multimodal data by incorporating a Spatial-Enhanced Feature Fusion Block to integrate 2D image features with estimated 3D spatial knowledge. The model achieves state-of-the-art performance on clinical datasets, demonstrating robust generalization to unseen surgical scenarios and downstream tasks without requiring additional annotations or sensor inputs.
You can now get SOTA 3D spatial reasoning in operating rooms from RGB images alone, thanks to a new vision-language model that fuses 2D and estimated 3D features.
Precise spatial modeling in the operating room (OR) is foundational to many clinical tasks, supporting intraoperative awareness, hazard avoidance, and surgical decision-making. While existing approaches leverage large-scale multimodal datasets for latent-space alignment to implicitly learn spatial relationships, they overlook the 3D capabilities of MLLMs. However, this approach raises two issues: (1) Operating rooms typically lack multiple video and audio sensors, making multimodal 3D data difficult to obtain; (2) Training solely on readily available 2D data fails to capture fine-grained details in complex scenes. To address this gap, we introduce Spatial-ORMLLM, the first large vision-language model for 3D spatial reasoning in operating rooms using only RGB modality to infer volumetric and semantic cues, enabling downstream medical tasks with detailed and holistic spatial context. Spatial-ORMLLM incorporates a Spatial-Enhanced Feature Fusion Block, which integrates 2D modality inputs with rich 3D spatial knowledge extracted by the estimation algorithm and then feeds the combined features into the visual tower. By employing a unified end-to-end MLLM framework, it combines powerful spatial features with textual features to deliver robust 3D scene reasoning without any additional expert annotations or sensor inputs. Experiments on multiple benchmark clinical datasets demonstrate that Spatial-ORMLLM achieves state-of-the-art performance and generalizes robustly to previously unseen surgical scenarios and downstream tasks.
