Search papers, labs, and topics across Lattice.
This paper presents a unified framework for real-time executional error detection in robot-assisted surgery by integrating multimodal inputs, including video, kinematic data, and descriptive textual prompts. The approach enhances the detection of technical errors by incorporating fine-grained contextual information about surgical activities and error types, which traditional methods often overlook. The proposed framework achieves significant improvements in error detection performance, with F1 score increases of up to 5% and 16.6% over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively.
Combining multimodal data with descriptive language boosts error detection in robot-assisted surgery by over 16%, highlighting the power of context in surgical precision.
Robot-assisted minimally invasive surgery improves surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. They also under-utilize complementary multimodal information. We propose a unified framework for executional error detection that leverages multimodal input, including video, kinematics, and descriptive textual prompts. Through activity prompting, we integrate descriptive language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual embeddings derived from vision encoders pretrained on surgical activity labels to compare the effectiveness of contrastive language-image embeddings with traditional image-based embeddings for error detection. By seamlessly integrating kinematic data with video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5\% and 16.6\% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively, we demonstrate the value of combining curated textual prompts with multimodal data for accurate error detection.
