Search papers, labs, and topics across Lattice.
This paper addresses the challenge of Generalized Referring Expression Comprehension (GREC) by introducing Refusal-Calibrated Group Relative Policy Optimization (RC-GRPO), a novel reinforcement learning approach that enhances Multimodal Large Language Models' (MLLMs) ability to refuse invalid object localization requests while maintaining high accuracy on valid ones. The method effectively balances the need for accurate localization with the ability to reject non-existent objects, which is crucial for preventing hallucinations in model outputs. Experimental results across three GREC benchmarks show that RC-GRPO significantly improves localization accuracy and refusal capability compared to existing methods like supervised fine-tuning and traditional reinforcement learning.
MLLMs can now refuse to localize non-existent objects without sacrificing their accuracy on valid requests, thanks to a novel reinforcement learning strategy.
We tackle the challenging yet underexplored task of Generalized Referring Expression Comprehension (GREC), which requires a model to localize the object described by a textual expression when it exists (positive sample) and to refuse output when it does not (negative sample). Although Multimodal Large Language Models (MLLMs) excel at localizing existing objects, they often fail to reject nonexistent ones due to the absence of negative samples during training, producing hallucinated bounding boxes. Existing post-training approaches such as supervised fine-tuning (SFT) and reinforcement learning (RL) enhance refusal behavior but usually degrade localization accuracy on positive samples, undermining the model's core competence. To address this, we propose Refusal-Calibrated Group Relative Policy Optimization (RC-GRPO), a calibrated RL strategy that strengthens the refusal ability of MLLMs while preserving localization performance. It enforces"None"outputs in rollouts for valid advantage estimation on negative samples and applies a penalty to prevent over-refusal on positives, achieving a balanced trade-off between accuracy and reliability. A second-stage reasoning reinforcement further consolidates causal understanding and interpretability. Experiments on three GREC benchmarks demonstrate that RC-GRPO attains superior localization accuracy while maintaining strong refusal capability.
