Search papers, labs, and topics across Lattice.
This paper introduces Kaleido, a novel algorithm-hardware co-design that accelerates video diffusion transformers (vDiTs) by leveraging channel-wise spatiotemporal correlations in latent space. By implementing a lightweight channelwise reuse algorithm, Kaleido significantly reduces redundant computations while maintaining superior generative quality, achieving improvements of over 17 dB. Evaluations demonstrate that Kaleido can deliver up to 5.9 times speedup and 16 times energy savings compared to existing state-of-the-art accelerators.
Kaleido achieves a remarkable 5.9x speedup in video diffusion transformers by intelligently reusing computations based on latent space correlations.
Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.
