Search papers, labs, and topics across Lattice.
This paper investigates the adaLN-Zero conditioning mechanism in diffusion transformers (DiT) and identifies zero-initialization as the most critical factor driving its superior performance over adaLN. The authors propose an analysis-guided initialization strategy, adaLN-Gaussian, which enhances optimization efficiency, and introduce a new conditioning mechanism, SE-adaLN-Zero, inspired by an SE-like structure. Extensive experiments on four datasets, particularly ImageNet1K, validate the effectiveness and generalization of these improvements in both class-to-image and text-to-image generation tasks.
Zero-initialization emerges as the key driver behind the superior performance of adaLN-Zero in diffusion transformers, reshaping our understanding of conditioning mechanisms in image generation.
Diffusion transformer (DiT), a rapidly emerging architecture for image generation, has gained much attention. However, despite ongoing efforts to improve its performance, the understanding of DiT remains superficial. In this work, we delve into and investigate a critical conditioning mechanism within DiT, adaLN-Zero, which achieves superior performance compared to adaLN. Our work studies three potential elements driving this performance, including an SE-like structure, zero-initialization, and a "gradual" update order, among which zero-initialization is proved to be the most influential. Building on this understanding, we propose an analysis-guided initialization strategy, termed adaLN-Gaussian, which serves both as an empirical validation of our analysis and as a practical initialization method that consistently improves optimization efficiency. On the other hand, inspired by the SE-like structure, we introduce an improved conditioning mechanism called SE-adaLN-Zero. Extensive experiments following DiT on four datasets, especially on ImageNet1K demonstrate the effectiveness and generalization of adaLN-Gaussian and SE-adaLN-Zero. Beyond class-to-image generation, we also evaluate the generalization of the two improved methods on text-to-image generation.
