Search papers, labs, and topics across Lattice.
Selective Repulsive Knowledge Distillation (SRKD) is introduced to address the challenge of distilling knowledge from large fetal ultrasound foundation models into compact, mobile-friendly models. SRKD decomposes contrastive knowledge distillation into diagonal (matched pair alignment) and off-diagonal (inter-class confusion repulsion) components, forcing the student to learn architecturally native features. An 11.4M parameter student model trained with SRKD outperforms a 304M-parameter teacher model in zero-shot fetal ultrasound tasks, achieving 88.6% vs. 83.5% accuracy on HC18 biometry validity and 0.784 vs. 0.702 F1 score on brain sub-plane classification, while running in real-time on a mobile device.
A novel knowledge distillation technique allows a tiny 11M parameter model to not just match, but *beat* a 300M parameter foundation model on fetal ultrasound tasks, opening the door to real-time AI on handheld devices.
Fetal ultrasound AI could transform prenatal care in low-resource settings, yet current foundation models exceed 300M visual parameters, precluding deployment on point-of-care devices. Standard knowledge distillation fails under such extreme capacity gaps (~26x), as compact students waste capacity mimicking architectural artifacts of oversized teachers. We introduce Selective Repulsive Knowledge Distillation, which decomposes contrastive KD into diagonal and off-diagonal components: matched pair alignment is preserved while the off-diagonal weight decays into negative values, repelling the student from the teacher's inter-class confusions and forcing discovery of architecturally native features. Our 11.4M parameter student surpasses the 304M-parameter FetalCLIP teacher on zero-shot HC18 biometry validity (88.6% vs. 83.5%) and brain sub-plane F1 (0.784 vs. 0.702), while running at 1.6 ms on iPhone 16 Pro, enabling real-time assistive AI on handheld ultrasound devices. Our code, models, and app are publicly available at https://github.com/numanai/MobileFetalCLIP.
