Search papers, labs, and topics across Lattice.
This paper introduces Stitch-Inferencer, a real-time, model-agnostic framework that enhances endoscopic video segmentation and tracking by utilizing an explicit panoramic canvas to stitch together valid observations across frames. By replacing implicit memory with a panoramic view, the framework allows for the preservation of previously observed pixels, effectively expanding the field-of-view and improving the handling of occlusions without the need for extensive retraining of downstream models. Experimental results demonstrate significant performance gains in anatomy segmentation and tracking tasks while maintaining real-time processing speeds, making it a viable solution for intraoperative environments.
Stitch-Inferencer achieves real-time endoscopic video segmentation by transforming how we manage occlusions and field-of-view limitations, outperforming traditional methods without the need for extensive retraining.
Surgical video understanding is fundamental to navigation systems. Endoscopic perception is often hindered by a limited field-of-view and frequent instrument occlusions, making spatio-temporal context essential for robust inference. These challenges have motivated video models that aggregate information across frames. However, existing video models typically store past observations implicitly in learned feature representations, often requiring task-specific video training, substantial annotated data, and increased computational cost. We propose Stitch-Inferencer, a real-time, model-agnostic inference framework that replaces implicit feature memory with an explicit image-space panoramic canvas. By stitching valid observations across frames, Stitch-Inferencer preserves previously observed pixels in an online, instrument-free view, expanding the effective field-of-view and providing direct access to regions that are temporarily occluded or absent from the current frame. Downstream segmentation or tracking models are applied to a compact region of interest on the panorama, and their predictions are reprojected to the current frame, enabling existing models to exploit long-range context without retraining. Experiments on anatomy segmentation and point/box tracking demonstrate consistent improvements across diverse baselines while preserving real-time throughput. The stitching module alone runs at over 60 FPS, providing a practical inference-time solution to enhance endoscopic perception in computationally constrained intraoperative environments. Source code will be made publicly available.
