Search papers, labs, and topics across Lattice.
This paper addresses the challenge of fine-grained action understanding in operating room (OR) environments, where clutter and occlusions complicate analysis. By introducing an action-centric benchmark and a novel vision-only temporal model, the authors demonstrate that traditional scene graph methods fail to capture temporal dynamics effectively. Their approach significantly improves multi-role action recognition by aligning features from multiple views, outperforming existing graph-based methods in the process.
Traditional scene graph methods falter in capturing the temporal structure of OR activities, while a new vision-only model achieves superior performance in multi-role action recognition.
Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance, yet remains difficult due to clutter, occlusions, and limited sensing. The prevailing approach to model this environment is scene graphs as an interpretable representation of OR interactions. Converting their frame-wise relational predictions into temporally extended, fine-grained actions however, is challenging without explicit temporal modeling. To enable a principled temporal evaluation of current OR understanding methods, we introduce the first action-centric benchmark built on a publicly available ego-exocentric OR dataset by defining a fine-grained, multi-role action taxonomy and generating dense action segments via distillation from ground-truth scene graph state changes. Experiments on this benchmark show that current scene graph prediction methods struggle to model temporal structure, even when adding explicit modeling through Graph Neural Networks. We therefore introduce a vision-only temporal model that outperforms graph-based methods significantly when using all available egocentric video as input. Building on this model we also introduce a novel multi- to single-view feature alignment strategy that improves single-view performance on multi-role action recognition, mitigating the need for extensive egocentric video capture. Benchmark and code will be released upon acceptance.
