Search papers, labs, and topics across Lattice.
This paper introduces GIVE, an innovative method that enhances Vision-Language-Action (VLA) models by integrating human gesture understanding to improve robotic manipulation in Human-Robot Interaction (HRI). By employing a dual-pathway approach that combines visual overlays of hand skeletons with semantic descriptions of gestures, GIVE significantly improves intent grounding, especially in scenarios where language instructions are ambiguous. Experimental results show that GIVE increases target object recognition accuracy by 40% and boosts overall task success rates by 80%, demonstrating its effectiveness in real-world applications.
Gesture understanding can elevate robotic manipulation success rates by 80%, transforming how robots interpret human intentions.
Human communication is inherently multimodal, where language is often accompanied by non-verbal cues such as gestures to convey intentions. However, current Vision-Language-Action (VLA) models treat robotic manipulation as a pure text-driven task, overlooking the important role of gestures in Human-Robot Interaction (HRI). This often leads to inaccurate intent grounding and unreliable manipulation when language instructions are ambiguous or underspecified. To address this challenge, we propose GIVE (Gesture Intent via Visual-Semantic Enhancement), an effective approach that enhances pre-trained VLA models with human gesture understanding without architectural modifications. Specifically, GIVE incorporates gesture information through two complementary pathways: a visual pathway that overlays hand skeletons and fingertip rays onto robot observations for explicit object grounding, and a semantic pathway that generates high-level descriptions of human gestures and task instructions for robust intent grounding. By jointly leveraging visual and semantic guidance, GIVE enables VLA policies to better associate gestures with manipulation behaviors and adapt to dynamic interaction intents. In real-world HRI experiments, GIVE substantially outperforms the baseline, improving target object recognition accuracy by 40% and overall task success rate by 80%, while demonstrating strong robustness and generalization to unseen spatial layouts and diverse participants.
