Search papers, labs, and topics across Lattice.
InstructSAM is introduced as a framework for multi-instance segmentation guided by arbitrary instructions, formulating it as a set-structured query prediction problem. It uses an explicit reasoning-to-instance query interface to connect a vision-language model (VLM) with SAM, injecting learnable instance queries into the VLM and using a hybrid-attention mechanism for improved instance enumeration. Experiments demonstrate that a 2B-scale InstructSAM achieves strong performance on instruction-driven segmentation benchmarks, surpassing existing methods and SAM's agentic pipeline while enabling efficient single-pass multi-instance prediction.
InstructSAM equips SAM with high-level instruction understanding and compositional reasoning for multi-instance segmentation, all without modifying SAM's core architecture.
In this paper, we introduce InstructSAM, a unified and streamlined framework designed for multi-instance segmentation under arbitrary instructions. We formulates instruction-driven instance segmentation as a set-structured query prediction problem and propose an explicit reasoning-to-instance query interface that elegantly bridges a vision-language model (VLM) and SAM3. Specifically, a bank of learnable instance queries is injected into the VLM and contextualized with instruction and visual information, enabling each query to serve as an instance-aware slot. A hybrid-attention mechanism further promotes interaction among these queries, visual tokens, and instruction tokens, improving instance enumeration and reducing duplicate predictions. The resulting LLM-conditioned queries are projected into SAM3's detector query space to drive accurate multi-instance segmentation in a single forward pass. This design equips SAM3 with high-level instruction understanding, compositional reasoning, and instance-level set prediction without modifying its core architecture. To support training and evaluation, we further construct Inst2Seg, a high-quality and large-scale instruction-based instance segmentation dataset and benchmark that couples free-form instructions with instance-level masks. Extensive experiments show that only 2B-scale InstructSAM achieves strong results across complex instruction-driven and phrase-level referring segmentation benchmarks, outperforming prior end-to-end methods and SAM3's agentic pipeline while enabling efficient single-pass multi-instance prediction.
