Search papers, labs, and topics across Lattice.
This study investigates the role of nonverbal cues in customer interactions with a teleoperated humanoid robot in a retail environment, revealing that 15.3% of robot responses were initiated by nonverbal behaviors rather than spoken commands. The authors developed a real-time multi-person, multi-label recognizer to identify service-relevant nonverbal cues and integrated this with a dialogue framework that conditions LLM-based responses on these cues. The findings highlight the limitations of traditional audio-only dialogue systems and demonstrate a novel approach for proactive robot engagement based on user behavior.
Nonverbal cues trigger over 15% of robot interactions, challenging the dominance of audio-only dialogue systems in service robotics.
Service robots in retail stores increasingly rely on cascaded speech pipelines (STT-LLM-TTS), yet many customer-robot interactions are initiated or guided by nonverbal behaviors such as approaching, waving, pointing, or showing items. This paper studies such cues in a real-world store deployment with a teleoperated humanoid robot and shows that a non-negligible portion of robot turns are triggered by nonverbal behaviors rather than spoken input, revealing a limitation of audio-only dialogue systems. In a 6-day in-the-wild deployment, 15.3\% of robot utterances were initiated by users'nonverbal behaviors rather than spoken input. Based on an analysis of observed customer behaviors, we define a set of frequent, service-relevant nonverbal cues and develop a real-time multi-person, multi-label recognizer that runs online from video. We then propose a dialogue framework that conditions LLM-based utterance generation on recognized nonverbal cue tokens, and optionally leverages a vision-language model when items are shown, enabling proactive robot responses without hand-crafted rules. We evaluate the approach offline on nonverbal-triggered turns and demonstrate an online prototype that reacts to users'nonverbal cues in real time.
