Search papers, labs, and topics across Lattice.
This paper introduces S$^3$T, a novel self-supervised framework for continuous video state tracking that leverages temporal sampling density to enhance accuracy without requiring labeled data or a separate teacher model. By treating a denser view of video clips as a teacher, the method enables a sparse-view student to learn to match its next-token distribution, resulting in significant improvements in VSTAT accuracy across various configurations. The approach not only outperforms prior self-evolving methods but also demonstrates effective transferability to real-world video state tracking tasks, achieving notable performance gains on benchmark datasets.
Self-supervised learning can achieve state-of-the-art video tracking accuracy without any labeled data or additional inference costs.
We introduce S$^3$T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S$^3$T improves VSTAT accuracy by $+1.74$ as a single model, $+2.38$ with souping, and $+2.70$ with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by $+7.95$ on VSTAT-YouTube state-tracking questions and $+4.50$ on MVBench Action Count.
