Search papers, labs, and topics across Lattice.
This paper investigates the sensicality of sentences in existing semantically deviant datasets by comparing human and LLM judgments, both with and without provided contexts. The study reveals that humans generally perceive sentences as anomalous rather than nonsensical, suggesting existing datasets may not be as nonsensical as assumed. Furthermore, the research demonstrates LLMs' ability to generate plausible contexts that render anomalous sentences more sensible.
LLMs are surprisingly good at making sense of nonsense, generating plausible contexts even for sentences designed to be semantically deviant.
Nonsensical and anomalous sentences have been instrumental in the development of computational models of semantic interpretation. A core challenge is to distinguish between what is merely anomalous (but can be interpreted given a supporting context) and what is truly nonsensical. However, it is unclear (a) how nonsensical, rather than merely anomalous, existing datasets are; and (b) how well LLMs can make this distinction. In this paper, we answer both questions by collecting sensicality judgments from human raters and LLMs on sentences from five semantically deviant datasets: both context-free and when providing a context. We find that raters consider most sentences at most anomalous, and only a few as properly nonsensical. We also show that LLMs are substantially skilled in generating plausible contexts for anomalous cases.
