Search papers, labs, and topics across Lattice.
The paper introduces AdaCluster, a training-free adaptive clustering framework to mitigate the quadratic attention complexity in video diffusion transformers (DiTs). It uses angle-similarity-preserving clustering for queries and euclidean-similarity-preserving clustering for keys, optimizing for both compression and accuracy. Experiments on large video generation models show significant speedups (1.67-4.31x) with minimal quality loss, demonstrating the effectiveness of the approach.
Get up to 4x faster video generation from diffusion transformers without sacrificing quality, thanks to a new clustering method that slashes attention costs.
Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic similarity or fail to adapt to heterogeneous token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity-preserving clustering method to query vectors for higher compression, and designs a euclidean-similarity-preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.
