Search papers, labs, and topics across Lattice.
This paper introduces SAM-MT, an innovative framework for real-time multi-target video segmentation that overcomes the limitations of traditional methods which suffer from increased latency as target counts rise. By leveraging explicit queries for individual targets alongside a shared global context representation, SAM-MT achieves efficient processing without sacrificing performance. The system demonstrates remarkable speed, maintaining over 36 FPS for 10 targets while preserving the segmentation accuracy of its predecessor, Segment Anything 2 (SAM2).
SAM-MT decouples latency from target count, enabling real-time video segmentation that rivals single-target performance even with multiple objects in view.
Modern Video Object Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked attention to keep individual identities distinct from cross-target interference, and sparse memory for stable temporal evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust video segmentation performance.
