Search papers, labs, and topics across Lattice.
This paper introduces Segmentation before Answering (SegAnswer), a novel approach that replaces traditional bounding boxes with pixel-level segmentation masks in Multimodal Large Language Models (MLLMs) for enhanced visual reasoning. By focusing on fine-grained visual details, SegAnswer effectively isolates regions of interest from background clutter, leading to improved accuracy in visual-language tasks. Experimental results across various benchmarks demonstrate that SegAnswer not only enhances performance in visual reasoning but also excels in segmentation tasks, validating its effectiveness in pixel grounding.
Shifting from bounding boxes to pixel-level segmentation in MLLMs leads to significant gains in visual reasoning accuracy and segmentation performance.
Recent advancements in Multimodal Large Language Models (MLLMs) have evolved from static perception to interleaved visual-language reasoning, often referred to as ``thinking with images''. A basic operation in this reasoning process is to zoom in on regions of interest (often represented with bounding boxes) to acquire finer visual details. In this paper, we propose \textbf{Seg}mentation before \textbf{Answer}ing (SegAnswer), which shifts the unit of zoom-in from the popular bounding box to pixel-level segmentation mask. By employing fine-grained masks to isolate the target area from cluttered environments, segmented visual input yields a more precise region of interest, effectively filtering out redundant background and interfering objects. Furthermore, the discrete patches of segmented visual input align more seamlessly with how MLLMs structure visual tokens via positional embeddings. In experiments, we evaluate SegAnswer across diverse benchmarks, including high-resolution perception, general perception, and hallucination. It achieves consistent improvements and also exhibits considerable performance on segmentation tasks, validating its capability for reliable pixel grounding.
