Search papers, labs, and topics across Lattice.
The paper introduces Latent Inter-frame Pruning with Attention Recovery (LIPAR), a training-free framework to reduce the computational cost of video generation by identifying and skipping redundant latent patches across frames. LIPAR incorporates an attention recovery mechanism to mitigate visual artifacts caused by pruning, approximating the attention values of pruned tokens. Experiments show LIPAR improves video editing throughput by 1.45x, achieving 12.2 FPS on an NVIDIA A6000 without compromising generation quality.
Accelerate video generation by 45% without retraining, simply by pruning redundant latent patches and cleverly recovering attention scores.
Current video generation models suffer from high computational latency, making real-time applications prohibitively costly. In this paper, we address this limitation by exploiting the temporal redundancy inherent in video latent patches. To this end, we propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches. Additionally, we introduce a novel Attention Recovery mechanism that approximates the attention values of pruned tokens, thereby removing visual artifacts arising from naively applying the pruning method. Empirically, our method increases video editing throughput by $1.45\times$, on average achieving 12.2 FPS on an NVIDIA A6000 compared to the baseline 8.4 FPS. The proposed method does not compromise generation quality and can be seamlessly integrated with the model without additional training. Our approach effectively bridges the gap between traditional compression algorithms and modern generative pipelines.
CMU ML
NUS