Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Frame Forgetting Network (FFN), which optimizes Test Time Training (TTT) for long videos by limiting updates to only three frames, thereby reducing computational demands while maintaining temporal context. By implementing a surprise metric to adaptively adjust the effective window size during TTT, FFN addresses the inefficiencies of traditional methods that rely on larger sliding windows. The empirical results show that FFN effectively performs dense segmentation and video classification on videos up to three hours long, significantly outperforming previous approaches on shorter datasets.
TTT can be made efficient for hours-long videos by focusing on just three frames and adapting the effective window size based on new information.
Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute. We present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a surprise metric: how much new information the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFNs empirical effectiveness on dense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos.