Search papers, labs, and topics across Lattice.
This paper introduces Heterogeneous and Adept Snapshot Knowledge Distillation (HAS-KD) for 3D semantic segmentation, leveraging knowledge transfer from multi-modal models and multiple model snapshots to enhance performance without incurring high computational costs. The method employs Information-oriented Heterogeneous Distillation (IHD) to enable a uni-modal model to absorb complementary knowledge from a multi-modal teacher, while the Adept Snapshot Distillation (ASD) reduces training expenses by utilizing model snapshots as experts. The proposed approach achieves state-of-the-art results on the ScanNetV2 and S3DIS datasets, demonstrating its effectiveness and efficiency in 3D segmentation tasks.
Knowledge distillation can now harness the power of multi-modal models and model snapshots, achieving top-tier segmentation performance without the typical computational overhead.
Multi-modal fusion and multi-model ensembling are prevalent in enhancing the performance of 3D semantic segmentation. Despite the impressive performance, these methods either rely on auxiliary input signals or suffer from costly computational expense. To efficaciously enhance the segmentation performance without introducing intolerable costs, we propose to transfer the rich knowledge from the multi-modal model (i.e., point clouds and images) and multiple model experts to the point-cloudbased network through knowledge distillation. Specifically, we present Information-oriented Heterogeneous Distillation (IHD) to help the uni-modal model absorb the complementary knowledge from the multi-modal teacher. We design the Information-Oriented Filtering (IOF) strategy to select informative images from the continuous image sequence for multi-modal fusion. This practice can boost the performance of the multi-modal teacher, thus benefiting the learning of the student. Besides, as opposed to vanilla model ensembling that requires the separate training of each expert, we propose Adept Snapshot Distillation (ASD). ASD treats the freely available model snapshots generated during the training phase as multiple experts, which significantly reduces the training cost for model ensembling. For each expert teacher, it only provides supervision to the student in the class where it is adept. The resulting Heterogeneous and Adept Snapshot Knowledge Distillation, dubbed HAS-KD, attains state-of-the-art results on ScanNetV2 and S3DIS datasets. HAS-KD can be seamlessly integrated into contemporary 3D segmentation algorithms and bring considerable gains without introducing extra inference burdens. The code will be made publicly available upon publication.
