Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study explores the complex interplay between verbal and nonverbal behaviors in virtual agents, emphasizing that human-like interactions require more than just aligning these channels. By utilizing a taxonomy based on Ekman's framework, the authors investigate how large language models (LLMs) can effectively select incongruent nonverbal behaviors that reflect the nuances of social context and emotional states. The findings from a human-subject study reveal that contextually mismatched nonverbal behaviors significantly influence observers' perceptions, enhancing the realism of virtual human interactions.
Contextually incongruent nonverbal behaviors can enhance the realism of virtual agents, revealing hidden emotional states that align with human social dynamics.
Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate the content of the verbal channel. However, human nonverbal behavior is shaped by more than the content of the speech. It is also influenced by speaker roles, interpersonal relationships, social context, and the cognitive and emotional states of the interactants. As a result, the nonverbal channel may reinforce, weaken, qualify, or even contradict the verbal channel. It may also reveal internal states that are hidden or only indirectly implied in speech, including emotional"leakage"that may be incidental to the immediate interaction. Modeling this richer relationship between verbal and nonverbal behavior is important for designing virtual agents that exhibit realistic, human-like behavior. It is especially critical in training contexts that require nuanced social interpretation, such as counseling simulations involving virtual patients. Drawing on Ekman's framework of verbal nonverbal relationships, we propose a taxonomy of categories in which mismatches between verbal and nonverbal behavior can occur. We then examine alternative approaches for realizing these behaviors using large language models, focusing on whether LLMs can select contextually appropriate mismatched verbal and nonverbal behaviors from a given dialogue and social interaction context. Finally, we evaluate the resulting behaviors in a human-subject study, assessing whether context-driven nonverbal behavior, when embodied in a virtual human, produces the intended effects on observers.