Search papers, labs, and topics across Lattice.
This paper introduces MLT-Dedup, a novel framework for large-scale online video deduplication that leverages Multi-Level representations and spatial-Temporal matching to efficiently identify near-duplicate videos. By employing a Multi-Level Video Encoder (ML-VE) for both fine-grained and sparse embeddings, the framework enhances candidate retrieval and precision in matching through the innovative DiF-SiM module. Experimental results show that MLT-Dedup achieves a remarkable 91% reduction in online repetition rates while maintaining 90% precision, alongside a 5x increase in indexing capacity.
MLT-Dedup slashes online video repetition rates by 91% while boosting indexing capacity fivefold, revolutionizing deduplication efficiency.
The explosive growth of user-generated video content on online platforms is accompanied by the emergence of numerous near-duplicate videos--videos that are identical or highly similar but differ by partial edits. These duplicates degrade user experience and increase storage and bandwidth costs, making large-scale video deduplication a critical task. Existing video deduplication frameworks face a fundamental challenge in retrieving sufficient high-quality candidates under a limited index budget, as well as trade-offs between efficiency and precision. To address these issues, we propose MLT-Dedup, an efficient large-scale online video deduplication framework with Multi-Level representations and spatial-Temporal matching. Our approach employs a Multi-Level Video Encoder (ML-VE) to extract both fine-grained frame-level and sparse clip-level embeddings: sparse embeddings support efficient candidate retrieval, while fine-grained embeddings are loaded for precise pairwise matching. During matching, we introduce DiF-SiM, a Differential Feature-enhanced Similarity Module capable of locating duplicated temporal segments and providing reliable similarity evidence to support policy-driven deduplication decisions. Extensive experiments on a real-world large-scale platform demonstrate that MLT-Dedup reduces online repetition rates by 91% at 90% precision. Furthermore, our sparse retrieval design achieves a 5x increase in indexing capacity, enabling broader candidate coverage in real-world deployment.
