Search papers, labs, and topics across Lattice.
This paper introduces novel, fully initialization-dependent generalization bounds for overparameterized shallow neural networks with Lipschitz activations. The key innovation is a new peeling technique that allows the bounds to depend logarithmically on the network width, unlike prior work that scaled with the square root of the width. The derived bounds, based on the path-norm of the distance from initialization, are shown to be non-vacuous through empirical validation and are tight up to a constant factor via a lower bound analysis.
Overparameterized neural nets can generalize well even with many parameters, and this work provides tighter, initialization-aware bounds that scale logarithmically with network width, offering a more nuanced understanding of this phenomenon.
Overparameterized neural networks often show a benign overfitting property in the sense of achieving excellent generalization behavior despite the number of parameters exceeding the number of training examples. A promising direction to explain benign overfitting is to relate generalization to the norm of distance from initialization, motivated by the empirical observations that this distance is often significantly smaller than the norm itself. However, the existing initialization-dependent complexity analyses cannot fully exploit the power of initialization since the associated bounds depend on the spectral norm of the initialization matrix, which can scale as a square-root function of the width and are therefore not effective for overparameterized models. In this paper, we develop the first \emph{fully} initialization-dependent complexity bounds for shallow neural networks with general Lipschitz activation functions, which enjoys a logarithmic dependency on the width. Our bounds depend on the path-norm of the distance from initialization, which are derived by introducing a new peeling technique to handle the challenge along with the initialization-dependent constraint. We also develop a lower bound tight up to a constant factor. Finally, we conduct empirical comparisons and show that our generalization analysis implies non-vacuous bounds for overparameterized networks.
