Search papers, labs, and topics across Lattice.
This paper introduces DiSC, a hardware accelerator for diffusion models that leverages both cached token reuse (CTR) and sparsity-inducing softmax thresholding with sparsity mask reuse (ST) to reduce computational overhead. CTR eliminates redundant token computation by exploiting spatial variations in the latent difference across diffusion steps, while ST induces sparsity in attention by reusing generated sparsity patterns. The DiSC architecture employs a hash-based distribution over on-chip memory banks to efficiently handle the resulting hybrid dense-sparse workload, achieving significant speedups and energy savings compared to A100 and H100 GPUs.
Diffusion models can run 4.7x faster and use 68% less energy thanks to a new sparsity-aware hardware accelerator that reuses cached tokens and attention sparsity patterns.
Transformer-based diffusion models offer superior scalability and performance but suffer from high computational overhead due to the iterative nature and quadratic complexity of self-attention at high resolutions. In this paper, we propose DiSC, a resolution-scalable, sparsity-aware hardware accelerator. At the software level, DiSC introduces two algorithms: Cached Token Reuse (CTR), and Softmax Thresholding with Sparsity Mask Reuse (ST). CTR introduces a mechanism that translates spatial variations in the input latent difference across steps into a token-level reuse decision, effectively eliminating redundant token computation. ST induces sparsity in attention operations by reusing a generated sparsity pattern, leveraging temporal similarity to bypass costly prediction overhead. Together, these algorithms provide resolution-scalable computational benefits and yield a moderate sparsity and hybrid dense-sparse workload. To exploit this efficiently, we design a specialized hardware architecture and unified dataflow. This architecture avoids dedicated sparsity-handling components; instead, a hash-based distribution over on-chip memory banks allows DiSC to reuse its existing compute engines for sparse operations, efficiently exploiting the induced sparsity with minimal hardware overhead. Evaluated on DiT and PixArt-Sigma, DiSC achieves 3.47-4.74x and 2.48-3.50x speedups over NVIDIA A100 and H100 GPUs, respectively, with energy savings ranging from 46.4% to 68.1%.
