Search papers, labs, and topics across Lattice.
This paper introduces feature information dynamics, an information-theoretic framework to quantitatively analyze when specific features are generated during the diffusion process by connecting mutual information change rates to denoising loss gaps. The framework reveals spectral autoregression in pixel diffusion and exposes fundamental representational differences between various diffusion model architectures under class-to-mask-to-Canny conditioning. These findings suggest that ordered feature generation could be a beneficial training strategy for diffusion models.
Diffusion models reveal feature generation order quantitatively, exposing architectural differences and suggesting ordered training benefits.
Diffusion models generate data through a continuum of denoising problems, and are widely observed to reveal coarse structure before fine detail. Yet, this intuition is mostly empirical and qualitative. We introduce feature information dynamics, an information-theoretic framework for localizing when a feature is generated during diffusion. Using the I-MMSE identity, we connect the rate of feature mutual information change to a gap between optimal unconditional and feature-conditional denoising losses, yielding practical estimators for feature information density. We further develop a chained decomposition that separates shared from incremental information in a feature hierarchy. We use this framework first to quantitatively confirm spectral autoregression in pixel diffusion, and then to extend the analysis beyond frequency: under a class $\to$ mask $\to$ Canny conditioning chain, the per-feature information densities differ across pixel, SDVAE, VAVAE, and RAE, exposing fundamental differences between these representations and suggesting that ordered generation could be beneficial for training diffusion models. Our code is available at https://github.com/AI4Science-WestlakeU/feature-information-dynamics.
