Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for open-vocabulary 3D object detection (3D-OVD) that addresses the limitations of existing two-stage pipelines by enhancing both the discovery of novel objects and the robustness of model training. The authors employ a co-distillation strategy that utilizes Hungarian matching to ensure high-quality object discovery based on geometric consistency, structural objectness, and semantic certainty. Additionally, they implement a dual-guidance learning scheme that incorporates scene-awareness-guided uncertainty regularization and LLM-guided hierarchical alignment, resulting in significant performance improvements on benchmark datasets SUN RGB-D and ScanNetV2.
Achieving unprecedented accuracy in 3D object detection by refining both object discovery and model training through innovative dual-guidance techniques.
Recently, open-vocabulary 3D object detection (3D-OVD) has gained increasing attention for its ability to detect unseen objects in 3D scenes. Existing approaches typically adopt a two-stage pipeline that first discovers novel objects using foundation models and then trains a 3D-OVD model based on these discovered objects. Although effective, this pipeline often suffers from inaccurate localization and mismatched classification during the discovery stage, which subsequently limits the performance of the model training stage. To address these limitations, we advocate for improving both the reliability of novel object discovery and the robustness of model training, and propose an innovative framework. Specifically, for reliable discovery, our co-distillation strategy distills high-quality novel objects by applying Hungarian matching over a comprehensive score that incorporates geometric consistency, structural objectness, and semantic certainty. To enhance robust model training, we further propose a dual-guidance learning scheme, incorporating a scene-awareness-guided uncertainty regularization for the regression head and an LLM-guided hierarchical alignment for the classification head, effectively mitigating the negative effects of imprecise 3D bounding boxes and semantic ambiguity. Extensive experiments on SUN RGB-D and ScanNetV2 demonstrate that our method achieves significant performance gains over state-of-the-art approaches. Code is available at https://github.com/shangboyuan/Co-3DGT
