Search papers, labs, and topics across Lattice.
The authors trained autoregressive transformers on synthetic corpora where shape is a stable feature dimension to investigate if these models can achieve second-order generalization (overhypothesis induction) in early word learning. Despite achieving perfect exemplar retrieval (first-order generalization), the models failed to generalize shape as a defining feature for novel nouns, remaining at chance levels. Analysis suggests that the models rely on template matching rather than structured abstraction, highlighting a limitation of autoregressive sequence learning.
Even with perfect memorization of examples, autoregressive transformers fail to learn higher-order generalizations about word categories, suggesting a fundamental gap in how these models learn compared to children.
Background: Children do not simply learn that balls are round and blocks are square. They learn that shape is the kind of feature that tends to define object categories -- a second-order generalisation known as an overhypothesis [1, 2]. What kind of learning mechanism is sufficient for this inductive leap? Methods: We trained autoregressive transformer language models (3.4M-25.6M parameters) on synthetic corpora in which shape is the stable feature dimension across categories, with eight conditions controlling for alternative explanations. Results: Across 120 pre-registered runs evaluated on a 1,040-item wug test battery, every model achieved perfect first-order exemplar retrieval (100%) while second-order generalisation to novel nouns remained at chance (50-52%), a result confirmed by equivalence testing. A feature-swap diagnostic revealed that models rely on frame-to-feature template matching rather than structured noun-to-domain-to-feature abstraction. Conclusions: These results reveal a clear limitation of autoregressive distributional sequence learning under developmental-scale training conditions.
