Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of memorization versus generalization in neural networks, revealing that the delay in generalization is linked to radial inflation of hidden representations during cross-entropy optimization. By employing a radial-angular decomposition of activation-space dynamics, the authors propose a method to penalize radial inflation, which leads to anisotropic weight regularization and encourages angular updates that converge to flatter minima. Empirical results demonstrate that applying this norm penalty significantly accelerates the grokking process in modular arithmetic tasks, achieving up to 6x faster learning in MLPs and Transformers.
Penalizing radial inflation in neural networks can accelerate algorithmic generalization by up to 6x, transforming how we approach training dynamics.
Why do neural networks memorize algorithmic training data long before they generalize? We present a geometric case study demonstrating that, on tasks where generalization requires discovering structured low-dimensional circuits, the memorization-generalization delay is driven by radial inflation of hidden representations under cross-entropy optimization. We formalize a radial-angular decomposition of activation-space dynamics and derive three testable propositions: (i) that penalizing radial inflation induces anisotropic, data-dependent weight regularization; (ii) that it suppresses radial gradient energy below the isotropic random baseline, forcing predominantly angular updates; and (iii) that it biases convergence toward flatter minima. To empirically validate these propositions, we study a single-hyperparameter norm penalty that softly constrains activations to a sqrt(d)-radius hypersphere. On modular arithmetic, this penalty accelerates grokking up to 6x across MLPs and Transformers, and halves training steps for a 10M-parameter nanoGPT on 3-digit addition.
