Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of grokking in a manageable 11,856-parameter transformer model, Glimmer-1-Base, focusing on modular arithmetic tasks. By employing a multi-seed approach, the authors reveal that grokking is a fragile phase transition influenced more by training-set coverage than task structure, with specific environmental factors capable of flipping outcomes without changing the overall rate. The findings challenge prior narratives by emphasizing the importance of multi-seed evidence and controlled numerical environments in understanding grokking dynamics.
Grokking is not just a static property; it's a delicate phase transition that can be easily disrupted by minor changes in computational conditions.
Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs. We instead study a publicly released ~11,856-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic, small enough to enumerate its weights, attention, and full input-output map, and we measure grokking as a multi-seed rate rather than a single outcome. In this fully-tractable regime grokking is a conditional, fragile phase transition. It is gated by training-set coverage, whose threshold tracks output cardinality (the modulus) more than task structure, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay reproduces the Omnigrok inverted-U at 12K parameters, a positive control on the rate measurement. Grokking also sits on a numerical knife-edge: two perturbations of the floating-point environment -- CPU thread count (reduction order) and CPU-versus-GPU execution -- each flip a minority of same-seed outcomes without a detectable shift in the aggregate rate. Decomposition into sub-task specialists helps chiefly by making coverage cheap rather than by adding supervision. Methodologically, multi-seed control under a fixed numerical environment overturns three dramatic single-run narratives in our own data, each a seed confound. The unit of evidence for grokking must therefore be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.
