Search papers, labs, and topics across Lattice.
This paper introduces CHAI, a novel cross-inference caching mechanism for text-to-video diffusion models that leverages Cache Attention to attend to shared objects and scenes across latents. CHAI accelerates video generation by selectively reusing cached latents from semantically related prompts, enabling high cache hit rates and reducing the number of required denoising steps. Experiments demonstrate that CHAI achieves a 1.65x - 3.35x speedup compared to OpenSora 1.2 while preserving video quality, even with as few as 8 denoising steps.
Text-to-video generation gets a serious speed boost: CHAI slashes inference time by up to 3.35x without sacrificing video quality, thanks to its clever cross-inference caching.
Text-to-video diffusion models deliver impressive results but remain slow because of the sequential denoising of 3D latents. Existing approaches to speed up inference either require expensive model retraining or use heuristic-based step skipping, which struggles to maintain video quality as the number of denoising steps decreases. Our work, CHAI, aims to use cross-inference caching to reduce latency while maintaining video quality. We introduce Cache Attention as an effective method for attending to shared objects/scenes across cross-inference latents. This selective attention mechanism enables effective reuse of cached latents across semantically related prompts, yielding high cache hit rates. We show that it is possible to generate high-quality videos using Cache Attention with as few as 8 denoising steps. When integrated into the overall system, CHAI is 1.65x - 3.35x faster than baseline OpenSora 1.2 while maintaining video quality.
