Search papers, labs, and topics across Lattice.
This study explores the phenomenon of grokking in neural networks, where models initially memorize data before achieving strong generalization after extended training. By analyzing the relationship between representation geometry and grokking, the authors discover that dimensionality collapse consistently precedes the onset of grokking across various tasks. They introduce Geometric Dimensionality Regularization (GeomDR), which modifies the effective dimensionality of hidden representations and can accelerate grokking by up to 52 times compared to standard training methods like AdamW.
Dimensionality collapse is a key precursor to grokking, and manipulating representation geometry can dramatically speed up generalization in neural networks.
Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.
