Search papers, labs, and topics across Lattice.
This paper introduces TDVR, a novel training-free reasoning framework designed for zero-shot 3D visual grounding that effectively disambiguates ambiguous textual queries and infers accurate viewpoints. By constructing a semantic 3D scene graph and utilizing chain-of-thought reasoning, TDVR enhances the understanding of input queries and optimizes viewpoint selection, significantly improving the model's ability to distinguish similar objects. Experimental results on the ScanRefer dataset show that TDVR outperforms state-of-the-art methods by 15.25% and 14.46% in accuracy metrics, highlighting its robustness in challenging scenarios.
TDVR achieves a remarkable 15.25% improvement in zero-shot 3D visual grounding accuracy by tackling ambiguous queries and viewpoint deficiencies head-on.
Zero-shot 3D visual grounding aims to localize specific objects based on textual descriptions and 3D visual input. However, the effectiveness of existing methods is significantly hindered by the ambiguous query text and deficient viewpoints. To address these issues, we propose TDVR, a training-free reasoning framework that disambiguates the input text and infers accurate viewpoints for zero-shot 3D visual grounding. First, we construct semantic 3D scene graph from the detected instances in the 3D point cloud. Subsequently, we put the original query, appearance and spatial relationship descriptions into the LLM for fusion, thereby disambiguating the initial input. We leverage chain-of-thought reasoning to generate the structured representation of disambiguated query. Then taking the scene graph and structured query as input, we get the optimal view via viewpoint reasoning to solve the problem of missing viewpoints during grounding. Based on the obtained optimal viewpoint, we further discriminate the distracting objects, enabling the model with the ability to distinguish similar instances. After that, we match the category text and appearance images with the query by computing the similarity of feature vectors. Finally, the target object was identified by integrating the viewpoint score, confusion score, category score, and appearance score. Compared with previous methods, our TDVR has stronger capabilities in viewpoint reasoning, similar object discrimination, and ambiguous query understanding. Experimental results on the public ScanRefer dataset show that our method outperforms the existing state-of-the-art methods by 15.25% and 14.46% in Acc@0.25 and Acc@0.5 respectively, demonstrating the effectiveness of our TDVR in addressing ambiguous query text and deficient viewpoints.
