Search papers, labs, and topics across Lattice.
This paper demonstrates that Uniform-based Discrete Diffusion Models (UDDMs) function as Associative Memories (AMs) capable of retrieving both seen and unseen data. They show that UDDMs form basins of attraction around data points through conditional likelihood maximization, enabling memory recall. By analyzing token recovery and conditional entropy, they identify a memorization-to-generalization transition governed by dataset size, offering a practical method to detect this transition in deployed models.
Language diffusion models aren't just generative, they're associative memories that reveal a sharp memorization-to-generalization transition detectable via conditional entropy.
When do language diffusion models memorize their training data, and how to quantitatively assess their true generative regime? We address these questions by showing that Uniform-based Discrete Diffusion Models (UDDMs) fundamentally behave as Associative Memories (AMs) $\textit{with emergent creative capabilities}$. The core idea of an AM is to reliably recover stored data points as $\textit{memories}$ by establishing distinct basins of attraction around them. Historically, models like Hopfield networks use an explicit energy function to guarantee these stable attractors. We broaden this perspective by leveraging the observation that energy is not strictly necessary, as basins of attraction can also be formed via conditional likelihood maximization. By evaluating token recovery of $\textit{training}$ and $\textit{test}$ examples, we identify in UDDMs a sharp memorization-to-generalization transition governed by the size of the training dataset: as it increases, basins around training examples shrink and basins around unseen test examples expand, until both later converge to the same level. Crucially, we can detect this transition using only the conditional entropy of predicted token sequences: memorization is characterized by vanishing conditional entropy, while in the generalization regime the conditional entropy of most tokens remains finite. Thus, conditional entropy offers a practical probe for the memorization-to-generalization transition in deployed models.
