Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces GoDeep, a novel approach to open-vocabulary 3D semantic segmentation that leverages a vision-language model to generate structured, entity-level descriptions from images without requiring a dedicated 3D training corpus or encoder. By projecting these descriptions into a general-purpose language-only embedding space, GoDeep achieves competitive performance on the ScanNet++ dataset and demonstrates superior fidelity in tracking out-of-vocabulary objects compared to traditional CLIP-based methods. The approach not only enhances localization of these objects but also ensures explainability at the point level, making it a significant advancement in 3D scene understanding.
Language-space embeddings in GoDeep outperform CLIP-based methods in identifying and localizing out-of-vocabulary objects, revealing a sharper distinction between presence and absence in 3D scenes.
Open vocabulary 3D semantic segmentation methods typically lift CLIP features into 3D. This embeds points in a joint vision-language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a vision-language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose, language-only embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that language-space embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that language-space embeddings separate presence from absence objects far more sharply than CLIP-based embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.