Search papers, labs, and topics across Lattice.
LVMT addresses long-term video segmentation challenges by introducing a GRU-based temporal propagation module that adaptively selects relevant object information and a Truncated Query Propagation (TQP) training strategy to mitigate memory and vanishing gradient issues during long video training. This enables effective tracking through occlusions and complex scenes by allowing for longer temporal supervision without prohibitive computational costs. LVMT achieves state-of-the-art performance across six benchmarks, outperforming prior methods by 10x in speed.
A novel training strategy allows video segmentation models to learn from hours-long sequences, overcoming memory and gradient limitations to achieve 10x speedup over SOTA.
Existing online video segmentation methods struggle to track objects in long, complex videos with long-term occlusions. We hypothesize that this limitation is caused by (i) the inability of their temporal propagation mechanism to adaptively select the object information that is propagated across time, and (ii) their inability to be trained on long videos due to memory requirements and vanishing gradients. To address the first limitation, we propose to use a lightweight GRU-based temporal propagation module that can learn to select which information it keeps in memory and propagates across time. Second, to allow training on long videos, we introduce Truncated Query Propagation (TQP), a training strategy in which the model processes a video in chunks of frames, where information about tracked objects is propagated between chunks but backpropagation is only conducted in individual chunks, enabling longer temporal supervision without out-of-memory issues, inference overhead, or vanishing gradients. The resulting model is called the Long-term Video Mask Transformer (LVMT). Extensive experiments on six benchmarks show that LVMT sets a new state of the art across a range of video segmentation tasks, while retaining the speed of the highly efficient model it is based on, making it 10X faster than the prior state of the art. Code: https://www.tue-mps.org/lvmt
