Search papers, labs, and topics across Lattice.
This paper analyzes the implicit bias of gradient descent (GD) when training shallow ReLU networks with squared loss on high-dimensional random features. It demonstrates that, in sufficiently high dimensions, the implicit bias of GD approximates the minimum-l2-norm solution among global minima. The approximation error is quantified as $\Theta(\sqrt{n/d})$, where *n* is the number of training examples and *d* is the feature dimension, bridging previous results that showed either non-existence or exact minimum-l2-norm solutions under more restrictive conditions.
ReLU networks trained with gradient descent surprisingly converge to near minimum-l2-norm solutions in high dimensions, even without orthogonal data.
Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work showed that the implicit bias does not exist in the worst-case (Vardi and Shamir, 2021), or corresponds exactly to the minimum-l2-norm solution among all global minima under exactly orthogonal data (Boursier et al., 2022). Our work interpolates between these two extremes and shows that, for sufficiently high-dimensional random data, the implicit bias approximates the minimum-l2-norm solution with high probability with a gap on the order $\Theta(\sqrt{n/d})$, where n is the number of training examples and d is the feature dimension. Our results are obtained through a novel primal-dual analysis, which carefully tracks the evolution of predictions, data-span coefficients, as well as their interactions, and shows that the ReLU activation pattern quickly stabilizes with high probability over the random data.
