Search papers, labs, and topics across Lattice.
This study revisits the volume hypothesis in deep neural networks, proposing that the size of loss-landscape basins correlates with generalization performance. By employing the Replica Exchange Wang-Landau algorithm, the authors investigate the relationship between training data size and the generalization advantage of stochastic gradient descent (SGD) compared to random weight sampling. The findings reveal that as training data size increases, the generalization benefit of SGD diminishes, providing a nuanced understanding of the conditions under which SGD excels.
As training data grows, the generalization edge of SGD over random sampling shrinks, challenging conventional wisdom about deep learning optimization.
Modern deep neural networks often contain far more parameters than needed to fit their training data, yet they achieve impressive generalization. A common explanation for this success is the implicit bias of stochastic gradient descent (SGD). An alternative volume hypothesis posits that, within low training-loss regions, loss-landscape basins leading to strong generalization occupy much larger regions of weight space than basins that generalize poorly, and therefore SGD is simply more likely to land in the former. Recent experimental explorations of this idea present seemingly contradictory results. While in one set of experiments randomly sampling the network weights until achieving zero training error yielded poor generalization, molecular dynamics density estimates supported the volume hypothesis. We observe that these experiments were performed at different dataset size regimes, and explore an intermediate regime using the Replica Exchange Wang-Landau algorithm to estimate the joint density of states over training and test accuracies in binary networks. Across several architectures and datasets, we show that the generalization advantage of gradient learning over random sampling training generally diminishes as the training data size grows, suggesting a resolution of the paradox.
