Search papers, labs, and topics across Lattice.
ActiveSAM transforms the Segment Anything Model 3 (SAM 3) into an efficient active-vocabulary segmenter for open-vocabulary semantic segmentation (OVSS) by leveraging a training-free, zero-shot inference framework. It employs a low-resolution presence preview to identify an image-specific subset of classes, significantly reducing the computational burden by decoding only the relevant classes at full resolution. This approach not only enhances the speed-accuracy tradeoff, achieving an average improvement of +1.4 mIoU over the state-of-the-art SegEarth-OV3, but also demonstrates robust performance under image corruption, making it ideal for real-world applications like autonomous driving.
ActiveSAM accelerates open-vocabulary segmentation by up to 5.5x while boosting accuracy, setting a new benchmark for efficiency in semantic segmentation tasks.
Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabulary semantic segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire dataset vocabulary, whereas each image contains only a small active subset of classes. We introduce ActiveSAM, a training-free, zero-shot inference framework that turns SAM 3 into an active-vocabulary segmenter. ActiveSAM first canonicalizes and expands class prompts, then estimates an image-conditioned active set from a low-resolution presence preview. Only the retained classes are decoded at full resolution, using bucketed prompt multiplexing with the frozen SAM 3 decoder. The preview stage uses only class-presence evidence and skips unnecessary segmentation-head computation, while the final stage applies margin-aware background calibration to suppress low-confidence pixels. ActiveSAM requires no target-dataset training, no weight updates, and no oracle class-presence labels. Across eight OVSS benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic segmentation, outperforming the current state-of-the-art SegEarth-OV3 by approximately +1.4 mIoU on average while running up to 5.5x faster on large-vocabulary datasets. ActiveSAM also demonstrates the strongest robustness under image corruption that simulates real-world distribution shift, making it well-suited for deployment in noisy-input domains such as autonomous driving and embodied AI. Code is available at https://github.com/VILA-Lab/ActiveSAM.
