Search papers, labs, and topics across Lattice.
The paper introduces SurgSTU-Pipeline, a deterministic pipeline for generating surgical video datasets with fine-grained spatial-temporal question-answer pairs. This pipeline addresses the lack of datasets that capture complex spatial-temporal dynamics in surgical videos, which are crucial for advancing computer-assisted surgery using VLMs. The authors create the SurgSTU dataset using this pipeline and demonstrate that fine-tuning VLMs on SurgSTU significantly improves their spatial-temporal understanding in surgical videos.
Current vision-language models can't handle the spatiotemporal complexity of surgical videos, but a new dataset shows how to fix it.
Surgical video understanding is a crucial prerequisite for advancing Computer-Assisted Surgery. While vision-language models (VLMs) have recently been applied to the surgical domain, existing surgical vision-language datasets lack in capturing and evaluating complex, interleaved spatial-temporal dynamics. Creating large scale datasets that accurately represent fine-grained spatial-temporal relationships in surgical videos is challenging due to costly manual annotations or error-prone generation using large language models. To address this gap, we introduce the SurgSTU-Pipeline, a deterministic generation pipeline featuring temporal and spatial continuity filtering to reliably create surgical datasets for fine-grained spatial-temporal multimodal understanding. Applying this pipeline to publicly available surgical datasets, we create the SurgSTU dataset, comprising 7515 video clips densely extended with 150k fine-grained spatial-temporal question-answer samples. Our comprehensive evaluation shows that while state-of-the-art generalist VLMs struggle in zero-shot settings, their spatial-temporal capabilities can be improved through in-context learning. A fine-tuned VLM on the SurgSTU training dataset achieves highest performance among all spatial-temporal tasks, validating the dataset's efficacy to improve spatial-temporal understanding of VLMs in surgical videos. Code will be made publicly available.
