Search papers, labs, and topics across Lattice.
The paper introduces Blackbird Language Matrices (BLM), a novel multiple-choice question answering task designed to evaluate linguistic competence in LLMs by assessing their ability to detect linguistic objects, systematic patterns, and the interplay between linguistic and reasoning errors. BLM datasets are constructed with multi-level structures within sentences, across input sequences, and within candidate answers, enabling targeted experiments on chunking and systematicity. Experiments demonstrate that LLMs can solve BLMs at varying performance levels, revealing their capacity to capture grammatical objects, attributes, and systematic patterns relevant to the task.
LLMs can be dissected like never before using Blackbird Language Matrices, a new task revealing if they're truly grasping linguistic structure or just pattern-matching.
This article describes a novel language task, the Blackbird Language Matrices (BLM) task, inspired by intelligence tests, and illustrates the BLM datasets, their construction and benchmarking, and targeted experiments on chunking and systematicity. BLMs are multiple-choice problems, structured at multiple levels: within each sentence, across the input sequence, within each candidate answer. Because of their rich structure, these curated, but naturalistic datasets are key to answer some core questions about current large language models abilities: do LLMs detect linguistic objects and their properties? Do they detect and use systematic patterns across sentences? Are they more prone to linguistic or reasoning errors, and how do these interact? We show that BLMs, while challenging, can be solved at good levels of performance, in more than one language, with simple baseline models or, at better performance levels, with more tailored models. We show that their representations contain the grammatical objects and attributes relevant to solve a linguistic task. We also show that these solutions are reached by detecting systematic patterns across sentences. The paper supports the point of view that curated, structured datasets support multi-faceted investigations of properties of language and large language models. Because they present a curated, articulated structure, because they comprise both learning contexts and expected answers, and because they are partly built by hand, BLMs fall in the category of datasets that can support explainability investigations, and be useful to ask why large language models behave the way they do.
