Search papers, labs, and topics across Lattice.
This paper introduces SA-VIS, a novel approach to video instance segmentation that leverages sparse frame annotations instead of the traditional dense annotations, which are costly in terms of compute and labor. By employing a Past-frames Feature Propagation (PFP) module that aggregates low-dimensional features from multiple frames, SA-VIS achieves performance that closely matches models trained on densely annotated data, with only a 0.4% drop in accuracy when using annotations for just 1/5 of the images. The method demonstrates significant improvements in performance on benchmark datasets such as YouTube-VIS, showcasing its effectiveness in reducing annotation requirements while maintaining high accuracy.
Sparse annotations can yield results nearly indistinguishable from dense ones, with SA-VIS achieving over 1% improvement in AP on state-of-the-art benchmarks.
Recent online video instance segmentation (VIS) methods have achieved impressive results, thus becoming the preferred approach to segment instances in videos. Despite the resurgence of impressive single image models, the online (or semi-online) VIS approaches outperform single-image models (e.g., based on SAM) by using long sequences of densely annotated frames during training. However,such a training setup of VIS is expensive in the sense of compute as well as dense annotations required. In order to solve these major flaws, we argue that the effective modeling of the instances and their evolution in videos do not require densely annotated frames. To that end, we propose a simple and effective module, called Past-frames Feature Propagation (PFP) which aggregates low-dimensional features from the image encoder of multiple frames. This simple low-compute module provides tremendous learning capability in using sparse video frame labels for end-to-end training. Combined with a light-weight frame-specific Instance Queries, our Sparse frame Annotation VIS (SA-VIS) significantly improves performance over its baseline. Most interestingly, our simple design that avoids complexities effectively bridges the gap in accuracy between training on sparsely and densely annotated video sequences. This translates to a mere 0.4% drop in performance of SA-VIS when using annotations for only 1/5 of the images in the dataset. Empirically, SA-VIS shows strong improvements over the baseline on YouTube-VIS 2019/2021/2022 and Occluded VIS (OVIS) and an over 1% improvement in AP on the state-of-the-art in a limited annotations scenario.
ETH