Search papers, labs, and topics across Lattice.
This study elucidates the phenomenon of delayed generalization, or grokking, by identifying a late-time relaxation mechanism in linear models trained with heavy-ball optimization and weight decay. The authors introduce the concept of the "grokking subspace," which is crucial for understanding how weight decay influences the slow decay of population risk and the timing of grokking. Their findings yield explicit predictions for grokking time and demonstrate the effects of different optimization strategies, validated through synthetic models and empirical observations in modular addition tasks.
The concept of the "grokking subspace" reveals how weight decay uniquely governs delayed generalization, challenging existing understandings of optimization dynamics.
Delayed generalization, or grokking, remains poorly understood despite extensive empirical study. We identify an exactly solvable late-time relaxation mechanism for grokking in linear models trained with full-batch heavy-ball optimization and weight decay, together with a locally quadratic extension to nonlinear neural networks. Our analysis reveals a distinguished population-active component of the empirical null space, which we call the grokking subspace. Along this subspace, the training predictions remain unchanged, leaving weight decay as the sole restoring force and giving rise to a slow dissipative relaxation governed by an exact discrete-time and continuous-time law. We show that only this subspace contributes to the slow asymptotic decay of the population risk and derive explicit iteration-scale predictions for the grokking time, recovering the familiar $(1-尾)/(畏位)$ scaling in the weak-regularization regime. The theory further predicts distinct effects of optimizer choice, distinguishing coupled $L_2$ regularization from decoupled weight decay, and yields causal predictions for interventions that modify the grokking component. We verify all theoretical identities without fitted parameters in a synthetic model where every subspace and relaxation rate is computable in closed form. We further observe genuine delayed generalization in modular addition, where the measured delay follows the predicted scaling and the late-time relaxation agrees closely with the theoretical clock.
