Search papers, labs, and topics across Lattice.
This study introduces EDITH, a novel robot framework that integrates verbal instructions with nonverbal signals such as gaze and gestures captured through smart glasses to enhance human-robot interaction. By employing a hierarchical policy structure, the system effectively interprets human intent from both language and visual cues, allowing the robot to execute tasks with minimal user input. Experimental results demonstrate that EDITH significantly reduces the cognitive load on users by enabling robots to respond to brief nonverbal signals, thereby improving interaction efficiency.
Robots can now interpret nonverbal cues alongside verbal instructions, drastically reducing the effort required from users to convey intent.
For natural human-robot interaction, a robot must understand human intent expressed not only through language but also through nonverbal signals such as gestures and gaze. However, current robot policies rely on language instructions as the sole interface for conveying intent, leaving nonverbal signals unused and placing the full burden of communication. In this work, we present EDITH, a robot framework that captures the human's nonverbal signals through continuous streams of first-person view and gaze from smart glasses, and uses them alongside language instructions as inputs to the robot policy. Our hardware system streams the human's first-person view, gaze, and speech to the robot in real time, transcribing the speech into language instructions. To handle these rich but noisy signals, we design a hierarchical policy in which a high-level policy infers the human's intent and produces a sequence of subtasks, where each subtask is represented as a fine-grained instruction paired with a keyframe that grounds the intent in the scene (e.g., the frame where the human points at the target object). A low-level policy then executes these subtasks. In our experiments on human-robot interactive tasks, EDITH enables the robot to act on the human's nonverbal signals even when intent is expressed only briefly, and significantly reduces user effort to convey intent compared to using language instructions alone. Visit our project page for source code and real-robot demo videos.
