Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a random-matrix model that analytically captures the transient regime of gradient flow in a linear teacher-student framework, where learning dynamics are influenced by anisotropic input covariance. The authors demonstrate that learning occurs when an isolated eigenvalue separates from a noisy bulk, leading to a transient Baik-Ben Arous-Péché (BBP) transition before overfitting takes over. Their findings reveal that the emergence and persistence of the teacher signal depend on the strength of the signal and the anisotropy of the covariance, providing insights into the mechanisms behind early stopping in training.
A transient spectral effect driven by anisotropy and noise reveals the hidden dynamics of early stopping in gradient descent training.
Empirical studies of trained models often report a transient regime in which signal is detectable in a finite gradient descent time window before overfitting dominates. We provide an analytically tractable random-matrix model that reproduces this phenomenon for gradient flow in a linear teacher--student setting. In this framework, learning occurs when an isolated eigenvalue separates from a noisy bulk, before eventually disappearing in the overfitting regime. The key ingredient is anisotropy in the input covariance, which induces fast and slow directions in the learning dynamics. In a two-block covariance model, we derive the full time-dependent bulk spectrum of the symmetrized weight matrix through a $2\times 2$ Dyson equation, and we obtain an explicit outlier condition for a rank-one teacher via a rank-two determinant formula. This yields a transient Baik-Ben Arous-Péché (BBP) transition: depending on signal strength and covariance anisotropy, the teacher spike may never emerge, emerge and persist, or emerge only during an intermediate time interval before being reabsorbed into the bulk. We map the corresponding phase diagrams and validate the theory against finite-size simulations. Our results provide a minimal solvable mechanism for early stopping as a transient spectral effect driven by anisotropy and noise.