Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of "natural ungrokking," where a language model initially learns a rule (such as pronoun-gender resolution) during pretraining but later forgets it despite the rule's presence in the training data. The researchers find that the survival of learned rules is determined by their support frequency in the training corpus, rather than the data-to-parameter ratio, with a predictable collapse pattern observed across different model scales and training conditions. Notably, the ability to control rule retention is asymmetric, as edits can eliminate rules but cannot restore them, revealing critical insights into the dynamics of rule learning and forgetting in language models.
Language models can forget learned rules during training, with their fate determined by how frequently the training data supports those rules, not by the model's size or complexity.
Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 925). By step 3,500 the same model scores near zero on the same probes, although the rule's evidence is still in the training data. We call this within-run reversal natural ungrokking: the corpus decides, with no trace in the loss curve, which learned rules a model keeps. Which rules survive is predictable from one corpus statistic: how often the training stream shows the rule winning. Across un-intervened runs (two corpora, three budgets, three seeds), support frequency decides a rule's fate; the data-to-parameter ratio only modulates how deeply a doomed rule falls. The same emerge-then-collapse dynamics appear in public Pythia checkpoints, collapse depth ordered by model scale as predicted. The forgetting is a displacement: a competing surface pattern out-competes the rule, and the log-probability margin between them crosses zero within 100 training steps of the behavioral collapse. Control over this fate is asymmetric: the same edit that destroys a rule on demand cannot restore it. Flipping support to counter-evidence in place kills the rule with monotone dose-response in two unrelated rules; but injecting support back, even to 450 times the level that naturally sustains it, buys no recovery. Every confirmatory threshold and prediction was pre-registered before the data it governed was read.