Search papers, labs, and topics across Lattice.
The paper introduces GeneralVLA, a hierarchical vision-language-action (VLA) model for robotics that leverages foundation model generalization for zero-shot manipulation and automated data generation. GeneralVLA employs an Affordance Segmentation Module (ASM) to perceive image keypoint affordances, a 3DAgent for task understanding and 3D trajectory planning, and a 3D-aware control policy for precise manipulation. Experiments across 14 tasks demonstrate that GeneralVLA outperforms existing methods like VoxPoser and generates data that leads to more robust behavior cloning policies without requiring real-world data or human demonstrations.
Achieve zero-shot robotic manipulation by guiding a hierarchical vision-language-action model with knowledge-guided trajectory planning, outperforming existing methods and eliminating the need for real-world data collection.
Large foundation models have shown strong open-world generalization to complex problems in vision and language, but similar levels of generalization have yet to be achieved in robotics. One fundamental challenge is that the models exhibit limited zero-shot capability, which hampers their ability to generalize effectively to unseen scenarios. In this work, we propose GeneralVLA (Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planning), a hierarchical vision-language-action (VLA) model that can be more effective in utilizing the generalization of foundation models, enabling zero-shot manipulation and automatically generating data for robotics. In particular, we study a class of hierarchical VLA model where the high-level ASM (Affordance Segmentation Module) is finetuned to perceive image keypoint affordances of the scene; the mid-level 3DAgent carries out task understanding, skill knowledge, and trajectory planning to produce a 3D path indicating the desired robot end-effector trajectory. The intermediate 3D path prediction is then served as guidance to the low-level, 3D-aware control policy capable of precise manipulation. Compared to alternative approaches, our method requires no real-world robotic data collection or human demonstration, making it much more scalable to diverse tasks and viewpoints. Empirically, GeneralVLA successfully generates trajectories for 14 tasks, significantly outperforming state-of-the-art methods such as VoxPoser. The generated demonstrations can train more robust behavior cloning policies than training with human demonstrations or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe GeneralVLA can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Code: https://github.com/AIGeeksGroup/GeneralVLA. Website: https://aigeeksgroup.github.io/GeneralVLA.
