Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates self-supervised learning on continuous video streams using strict sliding-window batches, deviating from standard i.i.d. image sampling. The authors identify high intra-batch similarity as a key challenge for existing methods like contrastive learning and MAE. They introduce StreamMAE, which adapts MAE with stream-aware regularization and motion-biased cropping to achieve performance comparable to i.i.d. training and ImageNet pretraining.
StreamMAE unlocks self-supervised learning on continuous video streams, matching i.i.d. MAE performance by tackling intra-batch similarity with novel regularization.
Self-supervised learning draws inspiration from infant visual development, yet standard training pipelines bear little resemblance to it: images are independently sampled and globally shuffled across epochs. We study self-supervised learning from continuous video streams, where frames are consumed in temporal order using strict sliding-window batches, without global reshuffling or multi-epoch replay. To this end, we construct WT++, a 95-hour urban walking-tour video dataset for streaming pretraining. Combined with a comprehensive evaluation suite we find that contrastive and distillation-based methods struggle in this setting, while MAE is more robust but still falls short of standard i.i.d. pretraining. We find that high inter-batch similarity, caused by sliding-window consumption across consecutive batches, does not explain this gap. The main challenge is high intra-batch similarity, where frames within each batch are near-duplicates. To mitigate this, we propose StreamMAE, which preserves the core MAE reconstruction objective while adapting the input pipeline with stream-aware regularization and motion-biased crop selection. StreamMAE outperforms streaming baselines, matches i.i.d. MAE trained on the same video data, remains competitive with ImageNet-pretrained MAE, and scales positively as the pretraining stream grows from 12 to 95 hours.