Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces xModel-KD, a cross-modal knowledge distillation framework for 3D point cloud segmentation that leverages 2D images to improve performance in data-scarce scenarios. The core innovation is a cross-modal fusion encoder trained with a contrastive objective to enforce feature consistency between 2D and 3D representations. Experiments on point cloud segmentation demonstrate a 2% mIoU improvement over LiDAR-only baselines, highlighting the benefits of cross-modal alignment for annotation-efficient 3D scene understanding.
Transferring visual cues from 2D images to 3D point clouds via cross-modal knowledge distillation boosts segmentation accuracy even when 3D labels are scarce.
Point cloud segmentation is a fundamental task in 3D scene understanding. Its progress is constrained by the high cost and time required for dense 3D annotations, making labeled samples difficult to obtain. Beyond annotation scarcity, different sensing modalities face inherent limitations. 2D images provide rich texture and appearance cues, yet they lack explicit depth and geometric structure. In contrast, 3D point clouds capture accurate spatial geometry but are sparse and contain no texture information. As a result, relying on a single modality restricts the richness of learned representations and weakens generalization. Although recent multi-modal methods that combine 3D point clouds with 2D images have demonstrated strong performance in tasks such as classification and retrieval, they typically depend on large-scale labeled datasets and have not been fully exploited for data-efficient dense prediction. To address these limitations, we propose a novel cross-modal knowledge distillation framework, xModel-KD, for 3D point cloud segmentation. Our method exploits the complementary strengths of 2D texture and 3D geometry by learning unified per-point representations through cross-modal alignment. Specifically, we design a cross-modal fusion encoder trained with a contrastive objective that enforces feature consistency between corresponding 2D and 3D representations across multiple views. By integrating powerful pre-trained backbones with a targeted fusion strategy, the proposed framework effectively transfers appearance cues from images to geometry-aware point features. Experimental results show that cross-modal fusion achieves a 2% absolute improvement in mIoU over a LiDAR-only baseline, demonstrating the benefit of leveraging complementary multi-modal information for scalable and annotation-efficient 3D scene understanding.