Search papers, labs, and topics across Lattice.
This paper addresses the challenge of Unsupervised Multimodal Semantic Segmentation (UMSS) by introducing the UniM2 framework, which leverages Cross Modal Correspondence Synergy (CMCS) to effectively integrate heterogeneous sensor data without the need for human annotations. The authors mitigate intermodal conflicts through a Cross Modal Harmonizer (CMH) that uses RGB as a stable reference, leading to significant improvements in segmentation performance. Experimental results on NYU Depth v2 and MFNet show that UniM2 achieves mIoU gains of 6.4% and 9.8%, respectively, outperforming existing methods in the field.
Unsupervised multimodal segmentation can achieve significant performance boosts without any human annotations, thanks to a novel framework that harmonizes sensor data.
Multimodal semantic segmentation (MSS) is essential for robust perception in complex environments, yet its potential remains largely untapped because of the prohibitive cost of human annotations. While unsupervised semantic segmentation (USS) has achieved strong results on a single RGB modality, its naive extension to multimodal data is often hindered by fusion degradation. This occurs because, without explicit supervision, existing frameworks struggle to reconcile the heterogeneous structural patterns captured by different sensors and therefore fail to effectively exploit their complementary information. In this paper, we make the first attempt to address the novel problem of Unsupervised Multimodal Semantic Segmentation (UMSS), aiming to effectively exploit complementary sensor information in a fully label free setting. To this end, we propose UniM2 (Unified Multimodal), a novel framework built on DINOv3 that transforms conventional fusion methods into consistent performance gains. Our key idea is to learn a unified latent space driven by Cross Modal Correspondence Synergy (CMCS) to extract intrinsic shared semantic cues, bypassing the need for label guided adaptive fusion. To mitigate inherent intermodal conflicts, we introduce a Cross Modal Harmonizer (CMH) that designates RGB as a stable reference, effectively suppressing inconsistent relational supervision while guiding the model to exploit complementary structural features. Extensive experimental results on NYU Depth v2 and MFNet show that UniM2 improves mIoU by 6.4% and 9.8%, respectively, demonstrating clear advantages over existing frameworks for UMSS.
