Search papers, labs, and topics across Lattice.
This survey paper comprehensively reviews knowledge distillation (KD) techniques applied to object detection, covering methods from CNN-based to Transformer-based architectures. It analyzes various KD algorithms used in object detection, including their novelty and limitations, and explores extended applications like improving lightweight models and enhancing small object detection. The survey also briefly touches upon KD applications in other domains such as image classification and semantic segmentation.
Knowledge distillation can make state-of-the-art object detectors practical for real-world deployment on edge devices, but the landscape of KD techniques is complex and rapidly evolving.
Deep learning models, especially for object detection have gained immense popularity in computer vision. These models have demonstrated remarkable accuracy and performance, driving advancements across various applications. However, the high computational complexity and large storage requirements of state-of-the-art object detection models pose significant challenges for deployment on resource-constrained devices like mobile phones and embedded systems. Knowledge Distillation (KD) has emerged as a prominent solution to these challenges, effectively compressing large, complex teacher models into smaller, efficient student models. This technique maintains good accuracy while significantly reducing model size and computational demands, making object detection models more practical for real-world applications. This survey provides a comprehensive review of KD-based object detection models developed in recent years. It offers an in-depth analysis of existing techniques, highlighting their novelty and limitations, and explores future research directions. The survey covers the different distillation algorithms used in object detection. It also examines extended applications of knowledge distillation in object detection, such as improvements for lightweight models, addressing catastrophic forgetting in incremental learning, and enhancing small object detection. Furthermore, the survey also delves into the application of knowledge distillation in other domains such as image classification, semantic segmentation, 3D reconstruction, and document analysis.
