Search papers, labs, and topics across Lattice.
Gaze2Act is introduced, a Vision-Language-Action (VLA) framework that incorporates human gaze as a dynamic intent signal for robot manipulation tasks. It bridges the ego-exo view gap by mapping first-person gaze to the robot's perspective, generating object masks and gaze points for precise target specification. Experiments on a Unitree G1 humanoid across 16 real-robot tasks show Gaze2Act achieves state-of-the-art performance in intent accuracy and task success, outperforming baselines in object disambiguation, fine-grained interaction, and dynamic intent steering.
Robots can now understand and act on your gaze while following language commands, enabling more intuitive and precise human-robot collaboration.
Vision-Language-Action (VLA) models have recently shown strong potential for robot learning by following language instructions. However, in practice, language alone is often insufficient to precisely convey human intent. It is difficult to describe which exact object to interact with among similar candidates, where to act on the object, or how the target may change during execution. To address this limitation, we propose Gaze2Act, a novel VLA framework that leverages human gaze as a dynamic and intuitive intent signal for complex interactive manipulation. Gaze2Act first bridges the ego-exo view gap by mapping first-person gaze into the robot's perspective through cross-view semantic matching, producing both an object mask and a gaze point for coarse-to-fine target specification. These cues are then integrated into the policy through perception-level prompting and action-level conditioning, allowing the robot to attend to relevant regions and execute precise interactions under dynamic intent. In a systematic evaluation across seven task categories and 16 real-robot tasks on a Unitree G1 humanoid, Gaze2Act achieves state-of-the-art performance in both intent accuracy and task success rate. It notably outperforms baselines in object disambiguation, fine-grained interaction, and dynamic intent steering. These results demonstrate that human gaze provides a natural, low-burden, and highly expressive modality for human-in-the-loop VLA control.
