Search papers, labs, and topics across Lattice.
This paper theoretically characterizes the topological configuration of the solution space in neural networks that leads to the phenomenon of delayed generalization, or grokking, during training. By analyzing the optimization dynamics of Adam with weight-shrinkage regularization, the authors reveal that solutions cluster in a shell-core structure, where generalization solutions are encased within memorization solutions. The study derives scaling laws for learning rate, batch size, and regularization, providing a deeper understanding of the conditions under which grokking occurs, supported by empirical validation.
Grokking isn't just a quirk鈥攊t's a structured phenomenon rooted in the topological properties of solution spaces shaped by optimization dynamics.
Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and $\ell_2$ regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.
