Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces LAViFiT, a novel framework that enhances the fine-tuning of vision-language models (VLMs) for recognizing surgical interactions by leveraging latent-action guidance. The method addresses the limitations of conventional approaches by employing an inverse dynamics model to capture visual changes from actions and a forward world model to focus on action-relevant regions, while a patch-level SIG Regularizer mitigates feature collapse. Experimental results demonstrate significant improvements in recognition accuracy and image-text alignment, with enhanced spatial coherence in the learned features related to instrument-tissue interactions.
LAViFiT achieves superior recognition of surgical interactions by ensuring that vision encoders focus on action-relevant regions, overcoming the limitations of traditional fine-tuning methods.
Understanding instrument-tissue interactions is essential for context-aware surgical AI and autonomous robotic surgery. Pretrained vision-language models (VLMs) and vision encoders offer an alternative to conventional interaction classifiers by transferring broad visual and semantic knowledge. However, adapting them to fine-grained surgical interactions remains challenging: (1) freezing the vision encoder depends entirely on pretrained representations that may retain noise and provide weak spatial localization, while (2) full fine-tuning can improve global semantic alignment without ensuring that the encoder learns meaningful features in the correct action region. We address these limitations by introducing LAViFiT, an end-to-end latent-action-guided framework for vision-language fine-tuning. An inverse dynamics model captures the visual changes induced by each action, while a forward world model drives the encoder to represent action-relevant regions. A patch-level SIG Regularizer further prevents local feature collapse without additional supervision, such as bounding boxes or pseudo-labels. Experiments across multiple encoders and datasets improve recognition and image-text alignment, while representation analyses show stronger grounding over the complete instrument-tissue interaction region and more spatially coherent features.