Search papers, labs, and topics across Lattice.
The authors construct STSG-VQA, an 18,458-pair benchmark that generates evidence-grounded temporal question answering by structuring surgical video into spatio-temporal scene graphs across object, event, and procedure levels. Explicitly modeling temporal dynamics addresses the severe blind spots of frame-centric supervision in surgical vision-language models where verifiable procedural continuity is critical. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B on this supervision yields micro accuracy gains of up to 24.39 percentage points over zero-shot baselines and up to 16.50 points over static scene-graph supervision across seven temporal categories.
Static frame-level scene graphs leave models blind to procedural dynamics, but grounding surgical queries across continuous spatio-temporal graphs drives an immediate 16.5-point accuracy jump over static supervision.
Despite recent advances in surgical vision-language models (VLMs), temporal reasoning remains limited because existing supervision is largely frame-centric. Frame-level scene graphs (SGs) have proven effective in providing structured representations of surgical environments but do not explicitly model the dynamics of surgical workflows. To explicitly model how surgical states evolve across time, we introduce a multi-level structured temporal supervision methodology that augments frame-level surgical SGs with object-level continuity, event-level interaction continuity, and procedure-level connectivity. We then execute temporal queries over the resulting spatio-temporal scene graphs (STSGs) to generate evidence-grounded question-answer pairs, which together form the STSG-VQA benchmark. Each question is linked to the temporal interval and STSG evidence used to derive its reference answer, enabling traceable verification. The benchmark contains 18,458 question-answer pairs across seven temporal categories. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B with STSG-derived supervision improves question-level micro accuracy by 24.39 and 19.56 percentage points over their zero-shot baselines and by 16.50 and 14.25 points over static scene-graph supervision, respectively. These gains span all temporal categories, indicating that STSG-derived supervision helps surgical VLMs reason over temporally grounded interactions rather than isolated frames. The code and dataset will be made publicly available upon acceptance.
