Search papers, labs, and topics across Lattice.
This paper investigates how large language models (LLMs) understand negation through both behavioral and representational systematicity. The authors find that while LLMs can recognize negation expressions to a degree via in-context learning, their performance is inconsistent and heavily influenced by output format, particularly struggling with negation scope recognition. Additionally, the study reveals that while function vectors can be effectively constructed for some negation tasks, challenges remain in reliably extracting them for scope recognition.
LLMs can grasp negation to some extent, but their ability to recognize negation scope is surprisingly inconsistent and format-dependent.
Understanding the meaning of negated sentences remains one of the challenges for language models, even in the era of large language models (LLMs). We analyze systematicity regarding LLM understanding of negation from two perspectives: behavioral systematicity and representational systematicity. For behavioral systematicity, we confirm that through demonstrations and in-context learning, LLMs can recognize negation expressions and scope within sentences to some extent, but they fail to achieve perfect performance. In particular, the difficulty of the negation scope recognition for models varies depending on the output format. For representational systematicity, we analyze the extent to which function vectors can be robustly constructed from in-context examples for tasks that are essential to understanding negation. The experiments suggest that while function vectors can be composed for negation cue extraction tasks, extracting function vectors for recognizing scope is more challenging.
