Search papers, labs, and topics across Lattice.
This study explores cross-architecture knowledge distillation (KD) from a fine-tuned DINOv2 Vision Transformer to a lightweight Visual State Space Model (LVSSM) for tea leaf disease classification, addressing the challenge of deploying accurate models on edge devices. By resolving training-stability issues and implementing a progressive convolutional stem and gated bidirectional selective-scan block, the authors achieve a significant accuracy improvement, raising test accuracy from 92.32% to 95.41% with a model that has 5.0 times fewer parameters than its teacher. The findings highlight the potential of KD in enhancing model performance while maintaining efficiency, making it viable for real-world agricultural applications.
Achieving a 3.09 percentage-point accuracy gain while reducing model size by 80% demonstrates the power of cross-architecture knowledge distillation in precision agriculture.
Automated tea leaf disease classification supports precision agriculture, yet deploying accurate models on edge devices remains challenging under tight compute budgets. Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment, while lightweight models trained from scratch on small agricultural datasets often underfit. We study cross-architecture knowledge distillation (KD) from a fine-tuned DINOv2 teacher (Vision Transformer) to a compact bidirectional Visual State Space Model (LVSSM) student, an underexplored direction because the architectures use fundamentally different token-mixing mechanisms. We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data: a single large patch-embedding convolution and a fusion layer that severs the residual path. With a progressive convolutional stem and gated bidirectional selective-scan block, the 4.45M-parameter student trains stably. Across three seeds, temperature-scaled logit distillation raises test accuracy from 92.32+/-2.14% to 95.41+/-1.17% (best single run: 96.20%; macro-F1: 94.45%), a +3.09 percentage-point mean gain. The student uses 5.0 times fewer parameters than the 22M-parameter teacher while retaining 98.3% of its accuracy. Ablations show that intermediate feature-alignment losses reduce accuracy, making simple logit-level KD the strongest configuration. A fair from-scratch comparison shows the gain is specific to students that start below the teacher. We report per-class metrics, confusion matrices, bootstrap confidence intervals, and FLOPs/latency measurements, and discuss limitations including the single-dataset scope and simplified non-official SSM implementation.
