Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates why deep models often assign higher density to simpler, out-of-distribution (OOD) data than to in-distribution data, a phenomenon termed the OOD anomaly. They decouple the trained network from the density estimator, using Jacobian-based and autoregressive self-estimators to analyze density across various models like iGPT, DINOv2, and diffusion models. The key finding is that deep networks consistently favor simple data, assigning higher densities to lower-complexity samples, even when trained on complex data alone, as quantified by Spearman rank correlation.
Deep networks don't just fail on out-of-distribution data; they actively *prefer* simpler, lower-complexity inputs, even after being trained exclusively on the most complex examples.
Estimated density is often interpreted as indicating how typical a sample is under a model. Yet deep models trained on one dataset can assign \emph{higher} density to simpler out-of-distribution (OOD) data than to in-distribution test data. We refer to this behavior as the OOD anomaly. Prior work typically studies this phenomenon within a single architecture, detector, or benchmark, implicitly assuming certain canonical densities. We instead separate the trained network from the density estimator built from its representations or outputs. We introduce two estimators: Jacobian-based estimators and autoregressive self-estimators, making density analysis applicable to a wide range of models. Applying this perspective to a range of models, including iGPT, PixelCNN++, Glow, score-based diffusion models, DINOv2, and I-JEPA, we find the same striking regularity that goes beyond the OOD anomaly: \textbf{lower-complexity samples receive higher estimated density, while higher-complexity samples receive lower estimated density}. This ordering appears within a test set and across OOD pairs such as CIFAR-10 and SVHN, and remains highly consistent across independently trained models. To quantify these orderings, we introduce Spearman rank correlation and find striking agreement both across models and with external complexity metrics. Even when trained only on the lowest-density (most complex) samples or \textbf{even a single such sample} the resulting models still rank simpler images as higher density. These observations lead us beyond the original OOD anomaly to a more general conclusion: deep networks consistently favor simple data. Our goal is not to close this question, but to define and visualize it more clearly. We broaden its empirical scope and show that it appears across architectures, objectives, and density estimators.