Search papers, labs, and topics across Lattice.
This paper introduces SPIRIT, a novel framework for recognizing surgical action triplets by learning spatio-temporal representations of instruments, verbs, and targets, while modeling their pairwise relations. The approach addresses the challenge of reliable triplet recognition across different surgical centers, where variations in video acquisition and surgical styles can impact performance. Evaluation on the newly established MultiBypass-4C-T40 dataset demonstrates that SPIRIT significantly outperforms existing methods, underscoring the importance of relational reasoning in multi-centric surgical contexts.
SPIRIT achieves superior surgical action triplet recognition by leveraging relational modeling, outperforming traditional methods across diverse surgical environments.
Fine-grained understanding of surgical activity is essential for context-aware assistance in the operating room, including safety monitoring, adverse event identification, and skill assessment. Surgical action triplets, defined as tuples of the form <instrument, verb, target>, provide a structured description of instrument-tissue interactions. A key open problem, however, is how to learn triplet representations that remain reliable across institutions, where surgical video varies in acquisition conditions, surgeon style, tool usage, and tissue handling, while existing triplet datasets do not support explicit evaluation of center-wise transfer. To address this problem, we propose \textbf{SPIRIT}, a structured framework for surgical action triplet recognition designed to learn interaction representations that transfer more reliably across centers. Instead of treating each triplet as a flat class label, SPIRIT first learns spatio-temporal representations for instruments, verbs, and targets, then models their pairwise relations, and finally composes them into coherent triplet predictions, with multi-head distillation used to stabilize learning. To evaluate this setting, we establish \textbf{MultiBypass-4C-T40}, a multi-centric dataset for dense surgical action triplet recognition in Roux-en-Y gastric bypass across four geographically distinct centers, with auxiliary phase and step annotations. Across multiple evaluation protocols, SPIRIT consistently outperforms strong recent baselines, highlighting the value of explicit relational reasoning for multi-centric triplet recognition. Code will be available at https://github.com/CAMMA-public/multibypass-4c-t40.
