Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper conducts a comprehensive audit of four prominent audio-visual synchronization metrics—AV-Align, ImageBind AV-relevance, JavisScore, and Synchformer/DeSync—using a rigorous reliability protocol. The findings reveal that while Synchformer/DeSync excels as a temporal-offset tracker, ImageBind and JavisScore better align with human perception metrics, highlighting significant discrepancies among the metrics themselves. The authors advocate for a more nuanced reporting approach, suggesting the use of Reliability Cards to convey metric-family breakdowns and confidence intervals instead of relying on a single synchronization score.
The analysis uncovers that existing AV-sync metrics not only disagree significantly but also fail to provide a reliable single score for audio-visual synchronization.
Automatic AV-sync metrics are widely used to rank and train audio-visual generators, but they are rarely audited as measurement instruments. We jointly audit AV-Align, ImageBind AV-relevance, JavisScore, and Synchformer/DeSync under a common reliability protocol: controlled-distortion monotonicity, preprocessing sensitivity, rank uncertainty, cross-metric agreement, PEAVS-proxy agreement, and learned fusion. The result is an axis split, not a single winner: Synchformer/DeSync is the strongest temporal-offset tracker ($\tau=0.84$), ImageBind/JavisScore better match the PEAVS human-aligned proxy ($\tau=0.20$) and content-disruption families, and AV-Align is the weakest standalone metric. The metrics mutually disagree (Krippendorff $\alpha=0.066$), and neither linear nor simple $k$-NN fusion improves PEAVS agreement over the best individual metric. We recommend reporting AV-sync as a Reliability Card (metric-family breakdowns with confidence intervals) rather than a single bare synchronization score.