Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Contextual Alternative Choice (CAC), a prompting method designed to quantitatively evaluate language models' acquisition of syntactic and discourse properties, specifically focusing on English determiners. CAC allows for direct comparison of language models against child language acquisition benchmarks and statistical baselines established in linguistic research. The study reveals that while no current models trained on comparable data match human children in both formal and functional aspects, some very large models approach human-level performance.
Language models still struggle to simultaneously master the formal syntax and functional discourse use of even simple words like "a" and "the," revealing gaps in their language acquisition compared to children.
We introduce quantitative metrics for child language acquisition to evaluate language models. Our focus is on the formal syntactic and functional discourse properties of determiners in English, which young children acquire early and accurately. We propose Contextual Alternative Choice (CAC), a new prompting method which provides targeted tests for both syntactic and discourse knowledge of language. The method enables direct comparison of language models against children, and more importantly, against statistical benchmarks independently established in empirical research. No current model trained on a comparable amount of data simultaneously meet both formal and functional benchmarks like human children, but some very large models do. We present our results as methodological and technical contributions, with specific emphasis on cognitive status of language models.