Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SAM-MT, an innovative framework for real-time interactive multi-target video segmentation that leverages the Segment Anything 2 (SAM2) model. By employing explicit queries for individual targets alongside a shared global context representation, SAM-MT effectively decouples processing latency from the number of targets, achieving over 36 FPS for 10 targets without sacrificing segmentation quality. The method incorporates decoupled masked attention and sparse memory to manage occlusions and maintain distinct identities, marking a significant advancement in VOS performance.
SAM-MT achieves real-time multi-target video segmentation at over 36 FPS, decoupling latency from target count while preserving high-quality segmentation.
Modern Video Object Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked attention to keep individual identities distinct from cross-target interference, and sparse memory for stable temporal evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust video segmentation performance.