Search papers, labs, and topics across Lattice.
This paper derives the eigenvalues of the Hessian matrix for linear neural networks across various configurations, addressing limitations in prior empirical and theoretical studies. The authors establish a direct relationship between the sharpness of the solution and the maximum proportion of samples in any class for classification tasks using MSE loss. Their empirical validation and systematic analysis reveal that these insights hold true even when relaxing previous assumptions, extending the findings to more complex learning scenarios.
The sharpness of neural network solutions is fundamentally tied to class distribution, revealing critical insights for optimization and generalization in deep learning.
The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second-order learning algorithms, etc. Prior works have focused on empirical results or pursued a theoretical treatment under overly simplified settings. In this work, we derive the eigenvalues of the Hessian of linear networks with arbitrary widths and depths, and datasets with an arbitrary number of samples, features, and labels. Importantly, for classification tasks with MSE loss, we identify that the sharpness of the solution is directly related to the maximum proportion of samples belonging to any class. We empirically validate our predictions and systematically analyze the effects of shedding the impractical assumptions one at a time, as well as incorporating nonlinearities. We observe that our predictions are considerably robust in most cases, allowing us to extend our conclusions to more practical learning setups.
