Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper analytically investigates grokking in a simplified model, disentangling the influence of initialization, signal strength, weight decay, sample size, and input dimension on distinct learning mechanisms: early fitting via nuisance features, core feature learning, and delayed generalization. The authors identify precise conditions for grokking, derive asymptotic expressions for generalization plateau lengths, and characterize early generalization and failure regimes, revealing the interpolation edge as a critical dynamical regime. These findings, supported by experiments, draw an analogy to chemical kinetics, suggesting grokking's lazy-to-rich transition reflects separated dynamical time scales.
Grokking's distinct phases—nuisance fitting, core learning, and delayed generalization—are analytically separable, with sharp conditions for occurrence and predictable plateau lengths.
In this work, we study grokking in a simple, analytically tractable setting whose rich phenomenology allows us to disentangle the effects of initialization, signal strength, weight decay, sample size, and input dimension. These factors can produce superficially similar learning curves through distinct mechanisms, complicating their interpretation. Our analysis distinguishes early fitting through nuisance features, core feature learning, and delayed generalization. We identify sharp conditions under which grokking provably occurs and derive analytic asymptotic expressions for the lengths of the generalization plateaus. Our results also characterize regimes of early generalization and failure to reach a prescribed test accuracy target. We show that the interpolation edge constitutes a distinct dynamical regime, where vanishing spectral gaps separate core learning from generalization, helping clarify the role of proximity to interpolation in existing accounts of grokking. The resulting picture is analogous to fast and slow phases in chemical kinetics and suggests that the lazy-to-rich transition discussed in the literature may reflect a similar separation of dynamical time scales. Our theoretical results are faithfully supported by extensive experiments.
Mila