Search papers, labs, and topics across Lattice.
This paper investigates the grokking phenomenon in modular arithmetic tasks using AdamW, revealing a "Spectral Gating" mechanism where generalization is gated by the optimizer's variance. The key finding is that grokking is constrained by a stability condition related to the curvature of the loss landscape, requiring sufficient gradient variance to overcome an "epsilon-stability threshold" and access the generalizing solution. Ablation studies identify three complexity regimes based on the number of parameters, highlighting the importance of anisotropic noise rectification by adaptive optimizers for generalization.
Grokking isn't about flat minima, but about AdamW's anisotropic noise unlocking access to sharp, generalizing solutions after a variance-driven "spectral gating" phase transition.
Standard optimization theories struggle to explain grokking, where generalization occurs long after training convergence. While geometric studies attribute this to slow drift, they often overlook the interaction between the optimizer's noise structure and landscape curvature. This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a ``Spectral Gating'' mechanism that regulates the transition from memorization to generalization. We find that AdamW operates as a variance-gated stochastic system. Grokking is constrained by a stability condition: the generalizing solution resides in a sharp basin ($λ_{max}^H$) initially inaccessible under low-variance regimes. The ``delayed'' phase represents the accumulation of gradient variance required to lift the effective stability ceiling, permitting entry into this sharp manifold. Our ablation studies identify three complexity regimes: (1) \textbf{Capacity Collapse} ($P < 23$), where rank-deficiency prevents structural learning; (2) \textbf{The Variance-Limited Regime} ($P \approx 41$), where generalization waits for the spectral gate to open; and (3) \textbf{Stability Override} ($P > 67$), where memorization becomes dimensionally unstable. Furthermore, we challenge the "Flat Minima" hypothesis for algorithmic tasks, showing that isotropic noise injection fails to induce grokking. Generalization requires the \textit{anisotropic rectification} unique to adaptive optimizers, which directs noise into the tangent space of the solution manifold.
