Search papers, labs, and topics across Lattice.
The paper introduces SemanticQA, a benchmark for evaluating language models' semantic phrase processing abilities, unifying diverse multiword expression (MwE) resources into a single testbed. It assesses LMs on extraction, classification, and interpretation tasks across lexical collocations, idiomatic expressions, noun compounds, and verbal constructions. Experiments reveal significant performance variations, especially in tasks requiring semantic reasoning, highlighting differences in LMs' comprehension of non-trivial semantic phrases.
LLMs still struggle to understand the meaning of common phrases, idioms, and compound words, revealing critical gaps in semantic reasoning.
We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates existing multiword expression (MwE) resources and reorganizes them into a unified testbed. It covers both general lexical phenomena, such as lexical collocations, and three fine-grained categories: idiomatic expressions, noun compounds, and verbal constructions. Through SemanticQA, we assess LMs of diverse architectures and scales in extraction, classification, and interpretation tasks, as well as sequential task compositions. We reveal substantial performance variation, particularly on tasks requiring semantic reasoning, highlighting differences in reasoning efficacy and semantic understanding of LMs, providing insights for pushing LMs with stronger comprehension on non-trivial semantic phrases. The evaluation harness and data of SemanticQA are available at https://github.com/jacklanda/SemanticQA.
Tsinghua AI