Search papers, labs, and topics across Lattice.
This paper introduces Rea2Seg, a two-stage framework that enhances image segmentation by leveraging multimodal large language models (MLLMs) for candidate mask discovery and comparative reasoning. By first identifying potential mask regions through attention maps and then scoring these candidates based on reasoning over complex queries, the framework effectively addresses the limitations of existing segmentation methods constrained by training data. The authors also introduce the ReasonSeg-SGDR benchmark, which evaluates models on a range of reasoning tasks, demonstrating that their approach significantly improves segmentation accuracy on this new benchmark.
Rethinking image segmentation, Rea2Seg achieves superior mask generation by combining candidate discovery with sophisticated reasoning over multimodal queries.
The rapid development of pretrained foundation models has enabled more general image segmentation. Multimodal large language models (MLLMs) have been widely explored for image segmentation with complex queries that require high-level reasoning. Despite promising progress, existing methods are often constrained by limited training data and the gap between MLLMs and mask generation modules. To better transfer MLLMs' perception and reasoning ability to complex reasoning-based segmentation tasks, we propose a two-stage framework Rea2Seg for mask generation and selection. Specifically, the framework first identifies potential regions as candidate masks based on the attention maps of a segmentation MLLM. It then employs an MLLM to reason over the question and candidate masks and assign scores to each mask. The final segmentation result is obtained by reranking the candidates and selecting the highest-scoring mask, reformulating image segmentation as candidate discovery followed by discriminative mask selection. We also notice that a large portion of questions in existing benchmarks focus on commonsense reasoning, and these questions usually do not fully require joint visual observation and reasoning. To address this issue, we introduce a new benchmark called ReasonSeg-SGDR that comprehensively evaluates a model's perception, grounding, and reasoning abilities across multiple dimensions, including discriminative recognition, spatial reasoning, geometric reasoning, and multi-step reasoning, with fine-grained mask generation. In addition, we collect training data to enhance MLLMs' ability to jointly understand multimodal queries and candidate masks, and to assign scores through reasoning. Experimental results on the proposed benchmark and ReasonSeg demonstrate the effectiveness of the unified mask generation and selection framework.
