Search papers, labs, and topics across Lattice.
This study introduces Hear2Act, a benchmarking protocol that evaluates how prosodic cues influence decision-making in task-oriented dialogues for text and spoken assistants. By systematically varying the presentation of user concerns through prosody versus explicit wording, the authors reveal that audio-capable LLMs significantly improve decision-making when they infer user concerns from audio cues rather than relying solely on explicit lexical feedback. The findings indicate that while prosody can enhance understanding, its effect diminishes when the same information is conveyed verbally, highlighting the nuanced role of prosody in dialogue systems.
Prosody can boost decision-making accuracy in dialogue systems by nearly 25% when it conveys user concerns that words alone cannot express.
Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions. We introduce Hear2Act, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes. For each scenario, we keep the task and user needs fixed while varying whether the same concern is conveyed explicitly in words or primarily through prosody, and evaluate decisions under transcript, audio, and concern-state access. Using Hear2Act, we evaluate two audio-capable LLMs. Under Prosody-mediated feedback, adding audio to the transcript changes the average optimal-solution rate only from 14.6% to 15.3%. In contrast, when models infer the concern status from audio, represent it in text, and use it for next-action selection, the rate rises to 39.6%, close to 40.7% with the ground-truth state. This contrast, however, largely disappears under Explicit lexical feedback, where the concern is verbally mentioned in the utterance. Together, these results show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do not reliably carry it into action without an explicit intermediate representation.
Amazon Science