Search papers, labs, and topics across Lattice.
This paper introduces FVAttn, an adaptive sparse attention system designed to enhance the efficiency of video generation in Video Diffusion Transformers by addressing the workload imbalance caused by adaptive Top-$p$ routing during multi-GPU execution. By implementing runtime load balancing and slack-aware sparse augmentation, FVAttn significantly reduces load imbalance and accelerates attention computation, achieving a 4.41x speedup over existing methods while maintaining competitive video quality. The results demonstrate that FVAttn not only improves execution efficiency but also optimizes resource utilization in high-resolution video generation tasks.
Achieving a 4.41x speedup in attention computation while maintaining video quality could redefine efficiency standards in video generation models.
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-$p$ routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present \method{}, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. \method{} uses Top-$p$ routing, a Top-$k$ safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, \method{} reduces average load imbalance from 1.34 to 1.08 and delivers a $4.41\times$ attention speedup over FlashAttention, while achieving a $2.02$--$2.11\times$ DiT inference speedup with competitive video quality.
