Search papers, labs, and topics across Lattice.
This paper investigates the mechanism behind grokking in neural networks, proposing that intrinsic task symmetries drive the transition from memorization to generalization. The authors identify a three-stage training dynamic: memorization, symmetry acquisition, and geometric organization, demonstrating that generalization emerges during the symmetry acquisition phase. They validate this account across algebraic, structural, and relational reasoning tasks, and introduce a symmetry-based diagnostic to predict and accelerate generalization.
Grokking isn't magic: it's all about neural nets learning to exploit the hidden symmetries baked into algorithmic tasks.
Grokking, the sudden transition from memorization to generalization, is characterized by the emergence of low-dimensional representations, yet the mechanism underlying this organization remains elusive. We propose that intrinsic task symmetries primarily drive grokking and shape the geometry of the model's representation space. We identify a consistent three-stage training dynamic underlying grokking: (i) memorization, (ii) symmetry acquisition, and (iii) geometric organization. We show that generalization emerges during the symmetry acquisition phase, after which representations reorganize into a structured, task-aligned geometry. We validate this symmetry-driven account across diverse algorithmic domains, including algebraic, structural, and relational reasoning tasks. Building on these findings, we introduce a symmetry-based diagnostic that anticipates the onset of generalization and propose strategies to accelerate it. Together, our results establish intrinsic symmetry as the key factor enabling neural networks to move beyond memorization and achieve robust algorithmic reasoning.
