Search papers, labs, and topics across Lattice.
This paper introduces GaussDet, a novel approach to 3D scene reconstruction that utilizes open-vocabulary 2D object detectors to enhance 3D Gaussian Splatting (3DGS) without relying on dense CLIP features. By learning instance features for individual Gaussians and employing a view-aggregation strategy, GaussDet effectively reduces noise in instance grouping and enables complex referential expression grounding. The method shows significant performance gains, achieving a 16.7% improvement in mean Intersection over Union (mIoU) for referential grounding in a zero-shot context, outperforming existing techniques.
GaussDet achieves a remarkable 16.7% boost in referential grounding accuracy, redefining the capabilities of open-vocabulary 3D scene understanding.
3D Gaussian Splatting (3DGS) has emerged at the forefront of 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has gained significant attention for real-world applications such as embodied AI. Recent methods achieve this by learning an instance feature attribute and assigning semantics by distilling high-dimensional Contrastive Language-Image Pretraining (CLIP) features directly into the scene representation. However, the instance grouping mechanisms of these methods either require a predefined number of instances or suffer from noise in their bottom-up grouping strategies. Furthermore, the reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referential expression grounding. We present GaussDet, a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities. We learn instance features for individual Gaussians to decompose the scene into 3D instance groups. By rendering these groups and aggregating semantic votes from multi-view 2D detections, we generate a robust View-Aggregated Semantic Label Distribution (VASD) for each 3D instance. This view-aggregation strategy acts as a strong regularizer, attenuating spurious labels caused by low-quality instance grouping. Our approach enables a straightforward, zero-shot extension from simple language queries to complex referential grounding. Extensive evaluations across two key tasks -- open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring expression grounding (Ref-LeRF) -- demonstrate that GaussDet achieves consistent improvements over existing methods. Most notably, we achieve a substantial 16.7% mIoU improvement in referential grounding within a strict zero-shot setting.
