Search papers, labs, and topics across Lattice.
This paper introduces CAMFusion, a novel multiview transformer architecture that fuses vision-language descriptors from multiple viewpoints into a unified per-3D-instance embedding for 3D semantic segmentation. They leverage multiview consistency as a self-supervision signal, significantly improving performance when combined with a standard supervised target-class loss. CAMFusion achieves state-of-the-art results on 3D semantic and instance classification benchmarks, including zero-shot evaluations on out-of-domain datasets, outperforming naive averaging or single-view descriptor selection.
Stop averaging and start attending: CAMFusion's cross-attentive fusion of multiview vision-language embeddings unlocks state-of-the-art 3D semantic segmentation, even in zero-shot scenarios.
Vision-language models have been key to the development of open-vocabulary 2D semantic segmentation. Lifting these models from 2D images to 3D scenes, however, remains a challenging problem. Existing approaches typically back-project and average 2D descriptors across views, or heuristically select a single representative one, often resulting in suboptimal 3D representations. In this work, we introduce a novel multiview transformer architecture that cross-attends across vision-language descriptors from multiple viewpoints and fuses them into a unified per-3D-instance embedding. As a second contribution, we leverage multiview consistency as a self-supervision signal for this fusion, which significantly improves performance when added to a standard supervised target-class loss. Our Cross-Attentive Multiview Fusion, which we denote with its acronym CAMFusion, not only consistently outperforms naive averaging or single-view descriptor selection, but also achieves state-of-the-art results on 3D semantic and instance classification benchmarks, including zero-shot evaluations on out-of-domain datasets.
