Search papers, labs, and topics across Lattice.
This paper introduces a novel analytic framework for determining the optimal early stopping time in neural network training using Rademacher complexity with L1-norm, circumventing the need for gradient descent training. By avoiding probabilistic assumptions common in existing methods, this approach provides a more relevant estimation of generalization error, particularly for linear models and linear regression. The findings demonstrate that the proposed method can effectively be applied to nonlinear neural networks, as evidenced by successful classification on the MNIST dataset.
Early stopping can be optimized without training, leveraging Rademacher complexity to enhance generalization in neural networks.
Training neural networks requires balancing the trade-off between fitting the training data and achieving robust performance on unseen inputs. This ability, commonly referred to as generalizability, is determined by the gap between the empirical risk on the training set (``empirical loss'') and the expected risk over the data distribution (``generalization error''). Existing approaches typically estimate the generalization error numerically, requiring gradient descent training and an ``early stopping''strategy. In this work, we introduce an analytic framework that estimates the optimal time of early stopping without the need for training. Several works in the literature also give such analytical estimations, but they are generally based on random matrix theory and often make assumptions on the distribution of the data or the eigenvalue distribution of the covariance matrix. In contrast, our work is based on Rademacher complexity (RC) without needing such probabilistic assumptions. For both theoretical and numerical reasons, it is more relevant to express RC with the L1- norm rather than with the L2-norm. We focus on the case of linear models and the problem of linear regression. Thanks to the ``linear probing''method, our results can, however, be successfully applied to nonlinear neural networks, as illustrated in the classification MNIST example.
