Search papers, labs, and topics across Lattice.
This study investigates how transformer models, specifically GPT-2 style architectures, respond to "impossible" language variants by evaluating their grammatical sensitivity and generative capabilities. The researchers found that while models maintained a gradual decline in grammatical sensitivity when exposed to perturbed languages, they significantly struggled with generating high-quality sentences as the length increased. These findings highlight generative deficiencies as a key factor explaining the models' biases towards human languages over unnatural ones.
Transformer models exhibit a striking generative deficiency, producing far fewer high-quality sentences in "impossible" languages, which could reshape our understanding of language model limitations.
Recent work suggests that transformer language models show a bias towards human languages over unnatural ("impossible") languages argued to be unacquirable by humans. However, this literature has largely based these claims on differences in sample efficiency and test-set perplexity, rather than on direct evaluations of the linguistic capacities that could plausibly explain non-attestation in human languages. We evaluate two theoretically motivated linking hypotheses: impossibility arising from deficiencies in grammatical sensitivity or generative production. Using GPT-2 style models trained on perturbed "impossible" variants of English, we measure sensitivity to grammaticality using BLiMP minimal pairs, finding that model performance exhibits only gradual degradation, mediated by the language's information locality. In contrast, these models exhibited pronounced failures in generation, producing substantially fewer high-quality sentences at longer lengths. Together, these results suggest generative deficiency and transmission failures as a plausible linking hypothesis between language model behaviour and non-attestation of impossible languages.
