Search papers, labs, and topics across Lattice.
This paper introduces OVIP-SG, a novel framework that integrates open-vocabulary perception into object-level 3D scene graphs, addressing the challenges of instance-level consistency and mapping fidelity in detecting small, fine-grained objects. By employing a vision-language model for scene-specific category enumeration and utilizing symmetric 3D Intersection over Union (IoU) association, OVIP-SG effectively preserves independent instances while partitioning scenes into functional search regions. The framework achieves significant improvements over existing methods, with a 6.31-point increase in class-mean accuracy and a 5.15-point boost in frequency-weighted mIoU, while also demonstrating practical effectiveness in real-world robotic applications.
OVIP-SG outperforms existing frameworks by preserving small object instances and enhancing retrieval accuracy, revolutionizing the mapping of fine-grained objects in 3D environments.
Integrating open-vocabulary perception into object-level 3D scene graphs is a double-edged sword. While vision-language detectors recover long-tail categories and small, fine-grained objects overlooked by closed-set models, they also tend to fragment large surfaces and merge small objects into larger neighboring objects, compromising instance-level consistency and undermining mapping fidelity. Moreover, existing methods struggle to retrieve previously unmapped targets or determine whether a queried object is absent, hindering robust embodied open-world navigation and exploration. We present OVIP-SG, a unified framework for instance-preserving semantic mapping, functional scene partitioning, and language-guided small, fine-grained object retrieval. OVIP-SG uses a vision-language model (VLM) to enumerate scene-specific categories for robust open-world detection. Symmetric 3D Intersection over Union (IoU) association and area-weighted feature fusion preserve small independent instances, while VLM-inferred object functions partition scenes into compact functional search regions. A four-stage cascaded retrieval pipeline further incorporates voxel voting and determines target absence from exploration coverage. Under a unified evaluation protocol on Replica, OVIP-SG outperforms ConceptGraphs by 6.31 points in class-mean accuracy (mAcc) and 5.15 points in frequency-weighted mIoU (F-mIoU) while achieving a class-agnostic native-instance Panoptic Quality (PQ) of 0.398. It reduces the search area to 21.8% of the indoor floor space and reaches 0.773 balanced accuracy for object-presence classification. Real-world robotic experiments further demonstrate its practical effectiveness.
