Search papers, labs, and topics across Lattice.
This paper introduces V-JEPA4A, a specialized variant of video self-supervised learning tailored for autonomous driving, which employs a saliency-driven masking policy to enhance feature representation learning. By focusing on semantically and temporally relevant cues in ego-centric driving videos, the method improves the predictive signal essential for downstream tasks, such as tracking and segmentation. The results show significant performance improvements, including a 25% reduction in identity switches on the BDD100k MOT dataset and achieving state-of-the-art metrics on Cityscapes and KITTI-2015 depth benchmarks with minimal additional training overhead.
Saliency-driven masking in video self-supervised learning can boost performance in autonomous driving tasks by preserving critical context that traditional random masking overlooks.
Video self-supervised learning through masked spatiotemporal prediction has emerged as a promising paradigm for learning feature representations from unlabeled data. However, existing methods typically rely on random masking, which indiscriminately removes regions irrespective of their semantic or temporal relevance. In ego-centric driving videos, this can weaken the pretext signal since safety-critical cues such as pedestrians, vehicles, lane boundaries, and dynamic interactions often occupy only a small portion of the frame, yet are central to downstream perception. We introduce V-JEPA4A, a domain-specialized variant of V-JEPA for autonomous driving that is pre-trained on publicly available driving videos with a novel saliency-driven masking policy. It accounts for semantically and temporally relevant context. The proposed policy preserves and predicts context according to semantic importance and temporal relevance, yielding more informative representation learning while retaining the efficiency of masked prediction. We evaluate the resulting encoders on four driving benchmarks spanning tracking, semantic segmentation, and depth estimation. The results demonstrate that V-JEPA4A reduces identity switches on BDD100k MOT by 25% over V-JEPA with random masking, achieves 73.2 mIoU on Cityscapes, and 3.75 RMSE on KITTI-2015 depth, while incurring only ~14% additional pre-training iteration overhead.
