Search papers, labs, and topics across Lattice.
This paper introduces DiT-Pruning, a novel post-training pruning method tailored specifically for Diffusion Transformers (DiTs) that addresses the challenges posed by their unique architecture and parameter distribution. By developing customized saliency criteria and a clustering-aware pruning granularity, the authors effectively balance the contributions of weights and activations, leading to improved identification of crucial model elements. Extensive evaluations demonstrate that DiT-Pruning maintains image quality even at high sparsity levels, achieving a mere 0.001 loss in CLIP score at 50% sparsity, significantly outperforming existing methods.
DiT-Pruning achieves unprecedented image quality retention in Diffusion Transformers, maintaining a CLIP score loss of only 0.001 at 50% sparsity.
Diffusion Transformers (DiTs) have demonstrated impressive performance in image generation but suffer from substantial computational overhead and resource consumption. Post-training pruning offers a promising solution; however, due to DiTs' unique architectural design and parameter distribution, traditional pruning methods are inapplicable, leading to significant performance degradation. Specifically, prior methods developed for LLMs, which derive metrics through a series of approximations, amplify the relative contribution of weights in the saliency metric. In addition, weights in DiTs exhibit significantly larger magnitudes than those in LLMs. Moreover, existing pruning granularity overlooks variations in model structures. In this paper, we propose DiT-Pruning, which improves pruning performance by introducing customized saliency criteria and pruning granularity. We design a novel metric that balances the contributions of weights and activations from an energy-based perspective, enabling more effective identification of important elements. Furthermore, we observe distinct clustering patterns in the two-dimensional weight space. Accordingly, we adopt a clustering-aware pruning granularity, enabling effective sparse allocation. Extensive evaluations on various DiTs show that our method consistently preserves image quality, especially under high sparsity. For FLUX.1-dev at 512x512 resolution on MJHQ, DiT-Pruning achieves only a 0.001 loss in CLIP score at 50% sparsity, dramatically outperforming recent pruning methods.
