Search papers, labs, and topics across Lattice.
This paper introduces EchoCache, an energy-guided cross-modal caching framework designed to enhance the efficiency of audio-driven video generation (A2V) by addressing misalignments in existing caching methods. By leveraging audio time-frequency energy as a saliency anchor, EchoCache optimizes latent-level cache updates and implements a dynamic timestep-latent caching mechanism, leading to significant improvements in latency and memory management. Experimental results demonstrate that EchoCache achieves a 2.46x speedup on the Wan2.2-S2V model while maintaining high-quality audio-visual consistency.
EchoCache achieves a remarkable 2.46x speedup in audio-driven video generation without sacrificing quality or coherence.
Audio-driven video generation (A2V) has achieved promising progress in synthesizing temporally coherent and audio-visually aligned videos, yet its inference remains expensive due to the iterative denoising process of diffusion models. Existing caching methods mainly exploit temporal redundancy in visual features while overlooking the cross-modal alignment of A2V, where audio drives visual generation with highly non-uniform temporal importance. In this paper, we identify two levels of misalignment in existing A2V caching methods: temporal-semantic and computation-storage misalignment. To address them, we propose EchoCache, an energy-guided cross-modal caching framework for efficient A2V generation. EchoCache leverages audio time-frequency energy as a saliency anchor to guide latent-level cache updates and further introduces a dynamic timestep-latent caching mechanism with quantized cache management for joint efficiency and memory optimization. Extensive experiments on mainstream A2V models show that EchoCache consistently improves the latency-quality trade-off while preserving generation quality and audio-visual consistency. In particular, on Wan2.2-S2V over the EMTD benchmark, EchoCache achieves a 2.46x speedup with the best overall performance. Code is available at https://github.com/IF-LAB-PKU/EchoCache.
