Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of grokking in transformers, revealing that the Frequency Synchronization Degree (FSD) serves as a reliable predictor of grokking events in modular arithmetic tasks. By analyzing nine configurations of modular addition, the authors demonstrate that FSD synchronizes significantly earlier than the grokking event itself, with a mean lead of 1,722 steps, and establishes a causal relationship between synchronization and improved generalization through regularization techniques. The findings indicate that the ability to predict grokking through FSD could enhance our understanding of transformer behavior and generalization capabilities.
Grokking in transformers can be predicted over 1,700 steps in advance using a novel metric, revealing a hidden structure in how these models learn.
Grokking -- where a transformer on modular arithmetic suddenly transitions from near-chance to near-perfect validation accuracy -- is attributed to a Fourier circuit, but its timing, causal structure, and controllability remain poorly understood. We introduce the Frequency Synchronization Degree (FSD), a normalised, permutation-tested metric for Fourier circuit synchronisation requiring no prior circuit knowledge. Across nine modular addition configurations (primes p in {53, 71, 97, 113, 131}, three seeds), FSD synchronises 500-3,000 steps before grokking (mean lead +1,722 steps; all nine positive, sign-test p~0.004), and precedes a restricted-logit loss baseline (Nanda et al.'s excluded loss) in all nine cases, making it the earliest available predictor. We provide direct causal evidence that the inter-phase gap is a regularisation phenomenon: forking training at the FSD-ceiling step and varying weight decay lambda produces strictly monotone earlier grokking, with Delta_t proportional to 1/lambda. This law replicates across three primes (p in {53,97,131}; R^2=1.00 and R^2=0.99 for two clean cases), captured as Delta_t ~ C/lambda, consistent with (1/lambda)*log(||W_mem||/tau). Architecture ablations show an attention-only model groks with a strong FSD precursor; an MLP-only model never groks; a single-layer model's FSD lags, confirming the precursor is a multi-block circuit property.