Search papers, labs, and topics across Lattice.
This study introduces DistillPath-KS16, a compact 22M pathology encoder that leverages knowledge distillation from larger, frozen pathology models to achieve performance close to that of much larger foundation models. By training on 6,000 public slides and utilizing only the final class and patch tokens from the teachers, the model achieves significant efficiency gains, running over 25 times faster than the top-performing Virchow2 model while maintaining competitive accuracy. The results demonstrate that even with fewer parameters, the distilled model outperforms existing smaller models and is task-dependent, highlighting the potential for efficiency in pathology encoding without sacrificing performance.
Achieving near state-of-the-art performance with a 22M encoder, DistillPath-KS16 runs over 25 times faster than its larger counterparts while retaining critical accuracy in pathology tasks.
Many high-performing pathology tile encoders are now foundation models with hundreds of millions to over a billion parameters. Encoding and storing the thousands of tiles in each whole-slide image with such models is costly on commodity hardware, so compact encoders that retain useful downstream performance are a valuable alternative. We present DistillPath-KS16, which starts from the existing 22M kaiko ViT-S/16 encoder and improves it by distilling from released pathology encoders used as frozen teachers. The recipe reads only the teachers' final class and patch tokens and trains on 6,000 public slides, needing neither their DINO nor iBOT pretraining heads nor a billion-tile corpus, so it applies to any released encoder that exposes backbone tokens. We distill four teachers spanning 86M to 1.1B parameters into the same student. Every variant improves the kaiko baseline on all three benchmarks we use, EVA, HEST, and PLISM, and the strongest teacher is task-dependent. On the seven-task EVA mean, DistillPath-KS16-Virchow2 reaches $0.795$, within $0.015$ points of Virchow2, the top-scoring model in our evaluation, at about $29\times$ fewer parameters; it also scores above H0-mini and GPFM on this aggregate metric, though that advantage is task-concentrated rather than uniform. Because it remains a 22M ViT-S/16 with 384-dimensional features, DistillPath-KS16 runs more than $25\times$ faster than Virchow2. Code is available at https://github.com/RamonKaspar/DistillPath, and released model weights are available at https://huggingface.co/collections/RamonK/distillpath.
ETH