Search papers, labs, and topics across Lattice.
This paper introduces Video Patch Pruning (VPP), a novel framework for efficient video instance segmentation that leverages temporal prior knowledge to enable early token reduction in Vision Transformers. VPP uses a differentiable temporal mapping module to select relevant patches in early layers based on foreground selectivity observed in deeper layers. Experiments on Youtube-VIS 2021 show VPP achieves up to 60% patch reduction with minimal performance drop (0.6%), surpassing image-based patch pruning methods.
Achieve 2x the efficiency of standard patch pruning by intelligently dropping video tokens in early transformer layers, guided by temporal context.
Vision Transformers (ViTs) have demonstrated state-ofthe-art performance in several benchmarks, yet their high computational costs hinders their practical deployment. Patch Pruning offers significant savings, but existing approaches restrict token reduction to deeper layers, leaving early-stage compression unexplored. This limits their potential for holistic efficiency. In this work, we present a novel Video Patch Pruning framework (VPP) that integrates temporal prior knowledge to enable efficient sparsity within early ViT layers. Our approach is motivated by the observation that prior features extracted from deeper layers exhibit strong foreground selectivity. Therefore we propose a fully differentiable module for temporal mapping to accurately select the most relevant patches in early network stages. Notably, the proposed method enables a patch reduction of up to 60% in dense prediction tasks, exceeding the capabilities of conventional image-based patch pruning, which typically operate around a 30% patch sparsity. VPP excels the high-sparsity regime, sustaining remarkable performance even when patch usage is reduced below 55%. Specifically, it preserves stable results with a maximal performance drop of 0.6% on the Youtube-VIS 2021 dataset.
