Search papers, labs, and topics across Lattice.
This paper introduces AV-SyncBench, a novel benchmark designed to independently evaluate temporal and semantic aspects of audio-visual synchronization, addressing the dimensional bias present in existing protocols. By utilizing a dataset of 3,269 videos across diverse scenarios, the benchmark allows for comprehensive assessment of feature extraction models in terms of alignment and downstream task performance. Key findings reveal significant differences in model performance when evaluated separately for temporal and semantic consistency, highlighting the necessity of decoupled evaluation in multimodal tasks.
AV-SyncBench reveals that existing benchmarks obscure critical differences in audio-visual model performance by conflating temporal and semantic evaluations.
Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for feature extraction models exhibit dimensional bias, typically focusing on either semantic matching or temporal offset detection. Moreover, their data construction remains coupled, preventing independent assessment of temporal and semantic consistency. We propose AV-SyncBench, the first benchmark to fully separate temporal and semantic evaluation for audio-visual synchronization. Built from in-the-wild videos, it spans Voice, Music, and Sound across 10 scenarios and 5 challenge tasks. Data are automatically filtered and manually verified to ensure on-screen sound sources. The benchmark contains 3,269 videos and 38,390 samples, and we evaluate five representative models to quantify feature quality for alignment and downstream tasks. The code and dataset are available at: https://fgt7t6g.github.io/AV-SyncBench.
