Search papers, labs, and topics across Lattice.
This paper explores the phenomenon of benign interpolation in deep learning, where models generalize well despite perfectly fitting their training data, challenging classical statistical learning theory. The authors argue that recent explanations invoking a simplicity preference among interpolating models create an explanatory gap, as they lack a provable connection to generalization. By contrasting these new accounts with classical theories that link model simplicity to generalization, the paper highlights the inadequacy of relying solely on the notion of simplicity without rigorous justification.
New interpretations of model simplicity in benign interpolation may obscure critical connections to generalization, leaving a significant theoretical gap.
Contemporary deep learning methods generalize well even when they fit their training data perfectly, a phenomenon known as benign interpolation. This phenomenon cannot be accounted for by classical statistical learning theory and has prompted a range of attempted new explanations in the statistics and machine learning literature. A common feature of these new proposals is an appeal to a simplicity preference among interpolating models, often presented as a form of Occam's razor. We clarify this debate for a philosophical audience and argue that this new appeal to simplicity creates an explanatory gap. The classical theory offers theorems which connect the simplicity of model classes to good generalization, thus underwriting methodological simplicity norms. The new accounts instead appeal to properties of individual models, which they interpret as a kind of simplicity. Lacking a provable connection to generalization, it is the name"simplicity"that does the work a theorem used to do, making a substantive and unargued assumption look like the application of a familiar methodological principle.
