Search papers, labs, and topics across Lattice.
This study evaluates the ability of large language models (LLMs) to recognize and update unspoken beliefs conveyed through implicatures and their cancellations. By creating the first expert-annotated dataset for this purpose, the authors reveal that LLMs significantly lag behind humans in understanding these nuanced communicative aspects, particularly in naturalistic contexts. The findings indicate that while LLMs can succeed in some belief updates, their performance is inconsistent and heavily influenced by the type and form of the implicature involved.
LLMs struggle to grasp unspoken beliefs, falling short of human-level understanding in recognizing and updating implicatures.
Human language is driven by unspoken beliefs and belief updates, making these critical to model for successful communication between large language models (LLMs) and their users. In this paper, we evaluate the ability of LLMs to recognize unspoken beliefs made through implicatures and to understand their updates through implicature cancellation: the pragmatic phenomenon whereby an utterance's implied meaning is weakened or negated. We create the first expert-annotated implicature cancellation dataset, [DatasetName], crowdsourced for human judgements of implicatures and their corresponding cancellations. We find that LLM belief update understanding lags behind that of humans, especially in more naturally-occurring scenarios. Additional control experiments suggest that successes in LLM belief updates may stem in part from a reliance on prior beliefs, and that failures in belief updates may depend on their type and on their form. Overall, our study suggests that current LLMs have not yet reached human-level understanding of unspoken beliefs and belief updates. Code and data are available at https://github.com/cesare-spinoso/ImplicatureX.
Mila