Search papers, labs, and topics across Lattice.
EfficientSync introduces a deformation-based framework for real-time lip synchronization that prioritizes the preservation of authentic textures from reference frames over traditional resynthesis methods. By employing a Dynamic Texture Mixer for channel-wise selection and Spatio-Temporal Shifted Adaptive Masking to minimize lower-face leakage, the approach achieves high visual quality and identity fidelity at 166 FPS on a single GPU. Experiments demonstrate that EfficientSync outperforms existing methods in both speed and texture authenticity, addressing critical limitations in current audio-driven lip synchronization techniques.
Achieving real-time lip synchronization at 166 FPS without sacrificing texture authenticity could redefine standards in video synthesis.
Audio-driven lip synchronization manipulates the mouth region of a talking-face video to match the driving audio while preserving head pose, identity, and background. Although the task is inherently local editing, prevailing approaches reconstruct the entire lower face with heavy GAN- or diffusion-based decoders, incurring substantial latency and, more critically, hallucinating intra-oral details such as teeth and lip wrinkles instead of preserving authentic textures. We contend that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain. We therefore present EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them. First, the Dynamic Texture Mixer reformulates multi-reference fusion as channel-wise selection, evaluating each spatially aligned reference in a global context and aggregating them by channel-wise weighted summation, preserving textural integrity at low cost. Second, Spatio-Temporal Shifted Adaptive Masking decomposes the source frame into lip-generation conditions and an independent background prior, suppressing lower-face leakage while blending the synthesized mouth seamlessly into the background. Third, STAR Sampling, a zero-overhead pre-processing step, retrieves the sharpest and most topologically diverse reference frames. Experiments on HDTF and VFHQ show state-of-the-art visual quality and identity preservation at 166 FPS on a single GPU. Video demos: https://alunaticat.github.io/EfficientSync/index.html.
