Search papers, labs, and topics across Lattice.
This study investigates whether large language models (LLMs) adequately capture the diversity of outputs found in their training data by analyzing the conditional entropy of model-generated outputs compared to that of the training data. The authors find that LLMs consistently produce outputs with lower conditional entropy than the training data, indicating a significant gap in output variability across various model families and tasks. To address this issue, they introduce a post-hoc correction mechanism that enhances output diversity while maintaining fidelity to the original model distribution, supported by a scalable optimization algorithm for implementation.
LLMs systematically underrepresent the diversity of their training data, with a notable conditional diversity gap that can be mitigated through a novel entropy-constrained projection method.
Large language models are trained to model conditional distributions over text, yet it remains inadequately understood whether they capture the full diversity of plausible outputs present in their training data. We study this question through an information-theoretic lens by comparing the conditional entropy of model-generated outputs with that of the corresponding training data. Given paired input-output samples, we use conditional entropy and its matrix-based analogue based on von Neumann entropy to measure output variability beyond what is explained by the conditioning input, without requiring multiple reference outputs for the same prompt. Across LLM families with publicly available training data, including OLMo, Pythia, and GPT-Neo, we consistently find that model-generated outputs exhibit lower conditional entropy than their training data, across different model scales, sequence lengths, and decoding strategies. We observe a similar conditional diversity gap beyond language modeling, including class-conditioned ImageNet generators and text-conditioned models trained on MS-COCO. To address this gap, we propose a post-hoc correction mechanism that generates multiple outputs for each input and reweights them through a matrix-entropy projection, increasing conditional diversity while remaining close to the original model distribution. We prove the concavity of the matrix-based conditional entropy functional, which makes the resulting entropy-constrained projection a convex optimization problem, and develop a scalable mirror-descent algorithm for its implementation. Our results reveal a systematic conditional diversity gap between modern generative models and their training data, and provide an information-theoretic framework for measuring and mitigating this gap.
