Search papers, labs, and topics across Lattice.
This paper introduces Bayesian information restricted diffusion (BIRD) models, which leverage restricted information about noisy data to elucidate how diffusion models can generalize from finite training sets without succumbing to memorization. By establishing an information-theoretic phase boundary that distinguishes between memorization and generalization based on mutual information and training data, the authors demonstrate that BIRD models can effectively approximate trained diffusion models early in training. Experimental results across various datasets validate the theoretical predictions regarding the transition point between memorization and generalization, highlighting the critical role of information restriction in generative AI.
Information restriction is the key to understanding how diffusion models can generalize rather than memorize, revealing a precise phase boundary that could transform generative AI practices.
How diffusion models circumvent the curse of dimensionality to learn complex distributions over high dimensional spaces from a finite training set, instead of memorizing it, remains a fundamental mystery. To address this, we introduce analytically tractable Bayesian information restricted diffusion (BIRD) models, in which each pixel observes restricted information about noisy data. A BIRD model time-reverses diffusion by inferring which past training sample produced its current restricted observation using the Bayesian posterior. This model class generalizes existing analytical diffusion models that use spatially local information restriction. We show that spatially local BIRD models closely approximate trained diffusion models \textit{early in training}, across different architectures such as UNets and DiTs. Under minimal assumptions on the data distribution, we identify an information-theoretic phase boundary between memorization and generalization in the joint space of amount of training data, time in the reverse generative process, and amount of information restriction: a BIRD model memorizes when the mutual information between its restricted noisy observations and the training data exceeds the log number of training points, and it generalizes otherwise. Experiments across a range of datasets confirm our theoretically predicted location for the transition. We find that generation proceeds near the edge of memorization: both spatially local BIRD models and early-training diffusion models track the memorization-generalization phase boundary by increasingly restricting information over time. Overall, our results reveal a fundamental role for information restriction in generative AI to circumvent the curse of dimensionality.
