Search papers, labs, and topics across Lattice.
This paper investigates the cognitive plausibility of transformer language models by comparing their predictions to human reading times in agreement attraction experiments. Eleven autoregressive transformers were evaluated on a diverse set of English syntactic configurations, focusing on prepositional phrase and object-extracted relative clause structures. The study reveals that while transformers align with human data in some cases (PP attachment), they fail to replicate human interference patterns in more complex object-extracted relative clauses, with significant divergence in predictions across different models.
Transformer language models stumble on complex syntactic structures, failing to mimic human-like error patterns in agreement attraction, suggesting current architectures lack crucial aspects of human morphosyntactic processing.
Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.
