Search papers, labs, and topics across Lattice.
This paper introduces DRAgent, a novel framework for Referring Expression Segmentation (RES) that leverages a discriminative reasoning approach to improve visual localization accuracy. By constructing a candidate space from detector outputs and employing a multimodal large language model (MLLM) for target selection, DRAgent mitigates issues related to localization bias and alignment errors commonly seen in previous methods. Experimental results indicate that DRAgent outperforms existing techniques on benchmark datasets like RefCOCO, RefCOCO+, and RefCOCOg, showcasing its effectiveness in producing precise pixel-level masks for specified objects.
DRAgent achieves superior localization accuracy by transforming the RES task into a discriminative reasoning challenge, significantly reducing bias and errors in object segmentation.
Referring Expression Segmentation (RES) aims to generate a pixel-level mask for the object specified by a language expression. Recent methods based on multimodal large language models (MLLMs) often rely on one-pass coordinate prediction for visual localization, which serializes continuous spatial locations as discrete text tokens and may lead to localization bias and alignment errors. To address these issues, we propose DRAgent, an MLLM-driven discriminative reasoning (DR) framework for RES. Instead of requiring the MLLM to generate localization coordinates, DRAgent first constructs a detector-generated candidate space and then uses the MLLM as a visual-semantic target discriminator. Specifically, the MLLM performs reliable target selection among potential distractors through a two-stage DR mechanism, which first screens high-recall candidates and then performs instance-wise verification. The selected target box is subsequently used as a spatial prompt for a foundation segmentation model to produce the final pixel-level mask. Furthermore, we construct a self-consistency-filtered reasoning-chain data pipeline for LoRA-based fine-tuning, providing more reliable supervision for enhancing the MLLM's discriminative reasoning capability. Experiments demonstrate that DRAgent achieves competitive performance on RefCOCO, RefCOCO+, and RefCOCOg.
