Search papers, labs, and topics across Lattice.
This paper investigates diversity bias in deep generative models by comparing the entropy-based diversity scores (Vendi and RKE) of generated samples with those of test data. The authors find that generated samples consistently exhibit lower diversity scores, indicating a systematic downward diversity bias. They attribute this bias to the finite-sample behavior of entropy estimators, where diversity estimates increase with sample size, leading to underestimation of true diversity during training.
Generative models systematically underestimate data diversity, losing up to X% of entropy, because training on finite datasets inherently biases them towards lower diversity estimates.
Deep generative models have achieved great success in producing high-quality samples, making them a central tool across machine learning applications. Beyond sample quality, an important yet less systematically studied question is whether trained generative models faithfully capture the diversity of the underlying data distribution. In this work, we address this question by directly comparing the diversity of samples generated by state-of-the-art models with that of test samples drawn from the target data distribution, using recently proposed reference-free entropy-based diversity scores, Vendi and RKE. Across multiple benchmark datasets, we find that test data consistently attains substantially higher Vendi and RKE diversity scores than the generated samples, suggesting a systematic downward diversity bias in modern generative models. To understand the origin of this bias, we analyze the finite-sample behavior of entropy-based diversity scores and show that their expected values increase with sample size, implying that diversity estimated from finite training sets could inherently underestimate the diversity of the true distribution. As a result, optimizing the generators to minimize divergence to empirical data distributions would induce a loss of diversity. Finally, we discuss potential diversity-aware regularization and guidance strategies based on Vendi and RKE as principled directions for mitigating this bias, and provide empirical evidence suggesting their potential to improve the results.
