Search papers, labs, and topics across Lattice.
This paper reformulates the evaluation of audio-driven talking-head generation as a sequence-alignment problem, addressing the inadequacies of frame-wise metrics that fail to account for natural timing shifts in speech. By integrating Soft Dynamic Time Warping into existing evaluation frameworks, the authors demonstrate that this new approach yields more stable and consistent results across various datasets. The findings reveal that temporally aligned metrics not only improve robustness to timing differences but also clarify the trade-offs between different modeling paradigms, such as synchronization versus realism.
Traditional frame-wise evaluation fails to capture the nuances of speech-driven facial motion, but a new sequence-alignment approach reveals clearer trade-offs in generative model performance.
Audio-driven talking-head generation has advanced rapidly, yet existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos. This assumption does not match speech-driven facial motion, which naturally includes slight timing shifts, different speaking speeds, and stylistic variations. As a result, conventional metrics may treat harmless timing differences as quality errors, making it harder to fairly compare methods and understand their trade-offs. In this work, we argue that evaluation of dynamic generative models should be formulated as a sequence-alignment problem rather than independent frame comparison. We introduce a unified sequence-level reformulation that integrates Soft Dynamic Time Warping into established evaluation pipelines. By aligning feature trajectories while preserving temporal order, the proposed framework provides robustness to bounded temporal misalignments without altering the underlying perceptual, identity, or synchronization encoders. We show that frame-wise evaluation can be viewed as a special case under rigid alignment, while sequence-level alignment provides improved stability, lower sensitivity to timing differences, and clearer separation between modeling paradigms. Building on this principled formulation, we conduct a large-scale benchmark of 20 methods across seven datasets spanning canonical, in-the-wild, and style-diverse scenarios under standardized protocols. Extensive experiments show that temporally aligned metrics are more robust to timing differences, provide more consistent results across datasets, and better reveal systematic trade-offs between modeling paradigms, such as synchronization versus realism and expressiveness versus stability.
