Search papers, labs, and topics across Lattice.
This paper introduces Self-Supervised Visual On-Policy Distillation (S^2VOPD), which innovatively creates an effective learning signal by subtracting information from the student rather than adding privileged data to the teacher. By leveraging asymmetric augmented views, S^2VOPD distills knowledge from a teacher model to a student model, achieving significant performance improvements across six fine-grained perception benchmarks. The method enhances the Qwen3.5-4B model's accuracy from 70.7% to 77.4%, outperforming all tested open-source models and recovering 96% of the gains typically associated with privileged information methods.
Subtracting information from the student rather than adding it to the teacher can yield performance improvements that rival those achieved with privileged data.
Visual on-policy distillation relies heavily on an informative teacher-student asymmetry, through either a larger, stronger teacher or privileged supervision, such as reference answers or ground-truth regions of interest. This raises a fundamental question: where can informative asymmetry come from when nothing privileged is available? We answer this by inverting where the asymmetry comes from. Rather than adding privileged information to the teacher, we subtract information from the student. This asymmetry creates the same effective learning signal for free as a teacher with access to information unavailable to the student, without ground-truth annotations, rewards, or a separate stronger teacher model. Building on this principle, we introduce Self-Supervised Visual On-Policy Distillation (S^2VOPD), a simple yet effective method that constructs on-policy learning signals from asymmetric augmented views. S^2VOPD distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image. We systematically explore a broad design space of visual augmentations and uncover that (1) asymmetry matters: all four augmentation families improve performance, while symmetric self-distillation degrades it; (2) strength matters: performance peaks at a moderate strength; and (3) the gap must remain task-consistent: augmentations that completely remove the question-relevant evidence can induce large but uninformative discrepancies. Across six fine-grained perception benchmarks, S^2VOPD improves Qwen3.5-4B from 70.7% to 77.4%, above all open-source models compared, up to Qwen3-VL at 235B, and surpasses GPT-5.4. While holding training data the same, it recovers 96% of the improvement achieved by methods with privileged information. Website is at https://williamium3000.github.io/s2vopd
