Search papers, labs, and topics across Lattice.
This paper introduces FrameBench, a benchmark designed to evaluate language models' ability to understand and utilize frame semantics in sentence comprehension. By testing models on their capacity to distinguish between different semantic frames evoked by the same verb in varied contexts, the study reveals that while smaller models struggle, several large language models outperform human reference scores. The findings highlight the nuanced capabilities of LLMs in implicit enrichment during comprehension, providing a new metric for assessing semantic understanding in AI.
Large language models can surpass human performance in understanding frame semantics, revealing their advanced implicit comprehension abilities.
In frame semantics, sentence comprehension is assumed to proceed by relating lexical meaning to background knowledge called semantic frames, thereby enabling readers to implicitly enrich the text with unstated information. Recent large language models (LLMs) have achieved strong performance across a wide range of downstream tasks. However, it remains unclear whether they can reproduce the kinds of implicit enrichment that humans naturally make during comprehension. To address this question, we introduce FrameBench, a benchmark grounded in frame semantics. FrameBench consists of multiple-choice questions that test whether models distinguish the frames evoked by the same verb across contexts. We construct the benchmark for English and Japanese using FrameNet-style resources and a generation-and-verification pipeline with native-speaker judgments. Our experiments on a diverse set of models reveal challenges for small models, while several large models surpass the human reference scores. We release the constructed FrameBench dataset and the code for dataset construction and evaluation at https://github.com/SasanoLab/FrameBench.
