Search papers, labs, and topics across Lattice.
This paper introduces SCOPE, a novel training-free sparse attention framework designed to enhance the efficiency of Diffusion Transformers (DiTs) by addressing the limitations of existing sparse attention methods. By implementing 3D-RoPE-aligned key subspace clustering and online per-head Top-$k$ estimation, SCOPE adapts to variations across heads and inputs, allowing for a more nuanced selection of keys that significantly improves both fidelity and latency. The results demonstrate that SCOPE achieves up to a 1.99脳 speedup on 720p HunyuanVideo while maintaining a PSNR of 28.46 dB, outperforming current training-free baselines.
SCOPE achieves a remarkable 1.99脳 speedup in video attention while enhancing fidelity, challenging the effectiveness of traditional sparse attention methods.
Diffusion Transformers (DiTs) incur quadratic self-attention cost over spatiotemporal tokens. Existing training-free sparse attention methods often construct sparse masks from block-level or cluster-level proxy scores, which can obscure fine-grained differences among keys and miss high contribution keys under aggressive sparsity. Moreover, such proxy scores may yield overly concentrated softmax distributions, causing Top-$p$ to retain too few keys for some query clusters. Although a fixed Top-$k$ minimum alleviates this failure mode, a shared value cannot adapt to variations across heads and inputs. To address both limitations, we propose SCOPE, a training-free sparse attention framework that combines 3D-RoPE-aligned key subspace clustering with online per-head Top-$k$ estimation for efficient video-DiT inference. SCOPE partitions post-RoPE keys into temporal, height, and width subspaces, clusters them independently, and aggregates the corresponding centroid scores through lookup tables to obtain per key proxy scores for each query cluster. Building on existing hybrid Top-$p$/fixed Top-$k$ selection, SCOPE derives a head-specific Top-$k$ value online by averaging the initial retained key counts within each head, weighted by query cluster size, and selects additional keys only for query clusters whose initial retained key counts fall below this value. Sparse attention is then computed over the selected original keys and values. Across six model--task configurations, SCOPE consistently outperforms existing training-free baselines in both fidelity and latency, achieving up to a $1.99\times$ end-to-end speedup on 720p HunyuanVideo with $28.46$ dB PSNR relative to dense attention.
