Search papers, labs, and topics across Lattice.
This paper introduces RefineRank, a novel approach for surgical spatio-temporal grounding that integrates a compact trainable module, RefineNet, to enhance the localization accuracy of candidate boxes generated by a frozen open-set detector. By combining language and regional features from a pre-trained medical vision language model, RefineNet predicts both a coordinate correction and a quality score for each candidate box, significantly improving the precision of object localization in surgical videos. The method achieves a state-of-the-art STG mIoU score of 0.421 on the MedVidBench dataset, demonstrating that effective box-level refinement can enhance performance without the need to retrain existing models.
A small, trainable module can dramatically improve localization accuracy in surgical video analysis, achieving the highest STG score on MedVidBench without retraining existing models.
Surgical spatio-temporal grounding (STG) requires locating, at each video time specified by a procedural question, the object that the question asks about. Existing approaches face a trade-off: vision language models understand the question context but produce imprecise coordinates, whereas open-set detectors provide localized candidate boxes whose confidence does not reflect which box answers the question. We introduce RefineRank, which closes this gap at the candidate-box level. A compact trainable module, RefineNet, combines the language and regional features of a frozen medical vision language model with the proposals of a frozen open-set detector: it predicts a bounded coordinate correction and a quality score for every candidate box, and a fixed decoding rule returns the original or refined box with the highest score. On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU, the highest displayed STG score, while its global multi-metric rank is 11. In a controlled evaluation on separate training and evaluation videos, coordinate correction raises the candidate oracle upper bound from 0.6772 to 0.7302, and ranking the joint pool of original and refined candidates by their RefineNet scores improves STG mIoU from 0.2719 to 0.4534, whereas separately trained selectors over the same pool reach at most 0.4186. These results show that a small box-level module can reconcile question understanding with precise localization without retraining either backbone. Code is available at [https://github.com/linzhe001/RefineRank](https://github.com/linzhe001/RefineRank).
