Search papers, labs, and topics across Lattice.
This paper investigates how transformer LMs learn linguistic categories by comparing their learning trajectories to abstract feature-based and concrete exemplar-based accounts of human language acquisition. Using divergence-based metrics to track next-token distributions during GPT-2 small training, the authors find that abstract class-level behavior emerges before lexical item-specific behavior. They also observe abrupt, sequential emergence of different linguistic behaviors, suggesting abstraction is crucial for language model learning.
Transformer LMs learn linguistic abstractions before memorizing specific lexical items, mirroring key aspects of human language acquisition.
Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human language acquisition that LMs may serve as an existence proof for.
Stanford HAI