Search papers, labs, and topics across Lattice.
To prevent teacher error propagation under domain shifts and noisy multimodal data, the authors formulate vision-language model distillation as an on-policy control problem that dynamically generates sample-wise training targets. The framework trains a lightweight controller via validation feedback to dynamically arbitrate between teacher logits, zero-shot priors, and hard labels using real-time student-teacher reliability and disagreement cues. This adaptive formulation consistently outperforms static vision-language distillation baselines on base-to-novel generalization and cross-dataset transfer tasks without incurring any test-time latency overhead.
Treating distillation targets as dynamic on-policy decisions rather than static teacher outputs prevents catastrophic error propagation when adapting compact vision-language models to out-of-distribution multimodal data.
Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in current vision-language distillation methods is typically constructed from the teacher prediction and applied uniformly to all training samples, making it unreliable under class and domain shifts. In this paper, we argue that distillation target construction should be treated as a dynamic training decision rather than a fixed recipe. To this end, we propose OnPoKD, an on-policy distillation framework for vision-language model adaptation. To the best of our knowledge, OnPoKD is the first framework that applies on-policy distillation to vision-language model adaptation by learning target construction as a policy decision. OnPoKD learns a lightweight controller that constructs sample-wise adaptive targets using reliability and disagreement cues from the teacher model, student model, and zero-shot prior. Instead of relying on a fixed teacher prediction, the controller dynamically balances teacher supervision, zero-shot prior guidance, and hard-label anchoring through bounded policy actions, allowing the distillation target to adapt to varying sample reliability and training stages. The policy controller is updated with validation feedback, encouraging target construction to optimize transferability rather than merely fitting the training distribution. Since the controller is only used during training, OnPoKD can be seamlessly integrated into existing vision-language distillation pipelines while preserving the original inference architecture and test-time cost. Extensive experiments on Base-to-novel generalization and Cross-dataset transfer benchmarks show that OnPoKD consistently improves over strong vision-language distillation baselines.
