Search papers, labs, and topics across Lattice.
This study introduces PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM) designed to enhance pointing-based visual grounding by integrating geometric reasoning with visual inputs. By leveraging a novel geometric reasoning pipeline and the EgoPoint-CoT dataset, the model effectively simulates human cognitive processes for interpreting spatial relationships, resulting in a significant 15.86-point improvement in mean Intersection over Union (mIoU) over traditional methods. The incorporation of an Adaptive Importance Weighting strategy further optimizes the learning process by dynamically adjusting reward signals based on training signal quality.
PointVG-R achieves a groundbreaking 15.86-point boost in mIoU by integrating geometric reasoning into visual grounding tasks, reshaping our approach to spatial interpretation in models.
Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images. In this study, we aim to mitigate the cognitive vulnerability of models in interpreting gestural spatial relations by proposing PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM). PointVG-R introduces geometric-aware reasoning for pointing-based grounding, enabling the model to think with images through the strategic integration of Reinforcement Learning (RL) and cold-start data. Specifically, we design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures. Furthermore, we construct EgoPoint-CoT, a high-quality visual Chain-of-Thought (CoT) dataset featuring detailed reasoning trajectories to guide the model via Supervised Fine-Tuning (SFT) and RL. To address the varying quality of learning signals encountered during training, we further propose an Adaptive Importance Weighting strategy based on Group Variance, which dynamically adjusts reward signals to optimize the learning process. Experimental results demonstrate that PointVG-R achieves SOTA performance, outperforming the baseline by $\textbf{15.86}$ points in mIoU. Extensive ablation studies further validate the efficacy of our proposed modules. Code: https://github.com/lingli1724/PointVG-R.
Tsinghua AI