Search papers, labs, and topics across Lattice.
This paper introduces RelGraphOV, a novel framework for open-vocabulary 3D scene understanding that leverages relationship-aware 3D scene graphs to enhance semantic segmentation beyond predefined categories. By utilizing vision-language reasoning to infer object relationships and employing an Adaptive Gated Dual-Stream Contextual GAT for feature aggregation, the method effectively prunes geometrically implausible connections without the need for manual annotations. Experimental results on multiple datasets, including ScanNetV2 and Replica, show significant improvements in performance and generalization capabilities compared to existing approaches.
Relationship-aware 3D scene understanding can significantly enhance open-vocabulary segmentation, outperforming traditional methods that ignore object relationships.
Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage
