Search papers, labs, and topics across Lattice.
This study systematically evaluates large language models' (LLMs) ability to infer pragmatic meaning from non-verbal responses in dialogue, a critical yet underexplored aspect of human communication. The findings reveal that LLMs exhibit a significant decline in accuracy鈥攗p to 60 percentage points鈥攚hen interpreting non-verbal cues compared to verbal ones, highlighting their limitations in recognizing indirect intent. Additionally, the research identifies specific failure modes and suggests that in-context learning can enhance LLMs' performance in interpreting non-verbal intent.
LLMs can miss up to 60% of the meaning when interpreting non-verbal cues, revealing a critical gap in their pragmatic understanding.
Although large language models (LLMs) have shown considerable progress in pragmatic language understanding, prior research has focused mainly on their comprehension of verbal behavior. Nonetheless, non-verbal behavior remains a fundamental component of human communication, especially when deliberately utilized in isolation to convey indirect meanings. In this work, we present the first systematic evaluation of LLMs' ability to infer pragmatic meaning in dialogue consisting solely of non-verbal responses. We explore three research questions: (1) Can LLMs recognize indirect intent conveyed through non-verbal responses? (2) When and how do LLMs fail to capture non-verbal intent? (3) How can we improve LLMs' ability to interpret non-verbal intent?. Through the evaluation, we observe that LLMs struggle to infer underlying meaning from non-verbal responses, with accuracy dropping by up to 60% points compared to verbal ones. Further extensive analysis reveals a behavioral pattern in LLMs' interpretations of non-verbal behavior and demonstrates that in-context learning facilitates pragmatic inference.
