Search papers, labs, and topics across Lattice.
This paper introduces iBKD, a novel knowledge distillation framework that effectively transfers convolutional inductive biases to Vision Transformers (ViTs) while preserving spatial grid structures. By employing an Inductive Bias Attention Module that utilizes learned weights and convolutional cross-attention, the method enhances the ViT's ability to leverage locality and translation equivariance during training. The results demonstrate that iBKD significantly outperforms existing distillation techniques across multiple benchmarks, particularly as the amount of training data decreases.
iBKD outperforms traditional distillation methods by preserving spatial grid structures, enabling Vision Transformers to excel even with limited training data.
Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting. The pooling, flattening, and logit-space projections it inherits from CNN to CNN pipelines discard the spatial grid in which locality and translation equivariance are encoded, and unlike a convolutional student, a ViT cannot rebuild that structure on its own. In this paper, we propose iBKD, a distillation framework that preserves the grid along the entire transfer path. Its core module, the Inductive Bias Attention Module, aggregates every student layer onto the teacher grid with learned weights, sharpens structural cues with channel and deformable spatial attention, and injects them through convolutional cross-attention that operates between grids rather than between token sets. The module is used only during training, so the deployed model is an unmodified ViT with no inference overhead. Across seven Transformer backbones and six data-scarce benchmarks, iBKD outperforms both locality-guidance methods and general knowledge distillation baselines, and its margin widens as training data shrinks.
