Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
The authors leverage the generative uncertainty of text-to-video (T2V) diffusion models to isolate semantically stable feature subspaces within pretrained video representations like VideoMAE and V-JEPA without human annotations. Addressing the high cost and ambiguity of spatio-temporal similarity labeling, the method exploits diffusion model consistency on familiar concepts to differentiate robust semantics from generative noise. Across video retrieval and out-of-distribution detection benchmarks, the resulting masked representation subspaces consistently outperform both the raw frozen embeddings and existing feature selection baselines.
Generative instability in text-to-video models is actually a feature, not a bug: measuring output variance across synthetic generations provides a zero-annotation signal to prune spurious spatio-temporal dimensions from foundation embeddings.
Evaluating semantic similarity between videos is a fundamental challenge in computer vision, essential for tasks ranging from out-of-distribution (OOD) detection to video retrieval. However, defining and labeling video similarity is notoriously difficult and expensive due to the complex spatio-temporal nature. In this paper, we propose a novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations. Our method is based on the observation that T2V models produce consistent outputs for familiar concepts but exhibit high variance and uncertainty when prompted with specialized concepts. We utilize this behavior to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA. Specifically, we learn a mask over these embeddings using purely generated data, encouraging the model to retain features that remain consistent across generations of general concepts while discarding those associated with generative noise or uncertainty. Experimental results across three key tasks demonstrate that our learned feature subspaces consistently outperform original pretrained features and baseline feature selection methods.