Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To resolve the visual ambiguity of coarse task-level instructions in Vision-Language-Action (VLA) fine-tuning, this work introduces frame-level visual guidance by overlaying human gaze crosshairs recorded during VR teleoperation directly onto camera inputs. At deployment, a lightweight auxiliary predictor infers gaze targets from recent observations and task descriptions, eliminating the need for runtime eye-tracking hardware or policy architecture modifications. Evaluating on $\pi_0$ across six real-world bimanual manipulation tasks, gaze prompting more than doubles mean task success from 26.3% to 56.0%.
Drawing predicted human gaze crosshairs directly onto input images more than doubles real-world bimanual VLA success (from 26.3% to 56.0%) without requiring runtime eye-tracking hardware or policy architecture changes.
Vision-Language-Action (VLA) fine-tuning pairs images with actions at every step, yet typically provides only a task-level language instruction, leaving moment-to-moment visual relevance implicit. We introduce \emph{eye-tracker-supervised gaze prompting}, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning. During training, recorded gaze locations are rendered as crosshairs on the robot's head-camera images. At deployment, a lightweight predictor estimates gaze locations from recent images and the instruction, supplying the same type of visual prompt without an eye tracker or changes to the policy architecture. Instantiated with $π_0$, gaze prompting increases mean success from $26.3\%$ to $56.0\%$ across six real-world bimanual manipulation tasks, with gains also observed when a single policy is trained on all six tasks. We release \textsc{GazeMani}, a dataset of $1{,}200$ teleoperated trajectories with synchronized gaze.
Tsinghua AI