Search papers, labs, and topics across Lattice.
TrackRef3D introduces a fully automatic pipeline for open-world referring segmentation in 3D Gaussian Splatting, eliminating the need for manual annotation and per-view pseudo masks. The core innovation is a multi-view consistent track-then-label paradigm, decoupling object discovery from semantic grounding via a Trajectory-Aware Semantic Consensus Module (TSCM). Experiments demonstrate state-of-the-art performance on benchmark datasets, showcasing the effectiveness of the approach.
Achieve open-world 3D segmentation without manual annotation by decoupling object discovery from semantic grounding.
Referring 3D Gaussian Splatting (R3DGS), which utilizes natural language for 3D object segmentation, has emerged as a crucial capability for embodied AI. However, existing methods typically rely on expensive per-scene manual annotation and per-view pseudo mask generation, which suffer from multi-view inconsistency and poor generalization to varying query specificities. To address this, we present TrackRef3D, a fully automatic pipeline that achieves open-world referring segmentation in 3D Gaussian Splatting (3DGS) without manual annotation by introducing a multi-view consistent track-then-label paradigm that fundamentally decouples object discovery from semantic grounding. Specifically, we propose a Trajectory-Aware Semantic Consensus Module (TSCM) which aggregates cross-view predictions via synonymous clustering and trajectory-aware voting to establish a canonical semantic identity, thereby ensuring multi-view consistency. Furthermore, we employ a visibility-aware description generation strategy to mitigate ambiguity and propose a Hybrid Training Strategy (HTS) that jointly optimizes coarse category semantics and fine-grained referential cues to ensure robustness under varying query specificities using a multi-positive contrastive objective. Extensive experiments on benchmarks demonstrate that TrackRef3D achieves state-of-the-art performance.
