Search papers, labs, and topics across Lattice.
The paper introduces Action with Visual Primitives (AVP), a VLA architecture where a VLM infers target visual primitives that condition a flow-matching action expert. This decouples high-level instruction comprehension and scene understanding from low-level motor control. AVP achieves a 27.61% improvement in success rate over baselines on real-robot pick-and-place tasks, demonstrating improved data efficiency and generalization.
Decoupling visual perception from motor control in robot learning yields a 27% performance boost and better generalization.
Vision-Language-Action (VLA) models have emerged as a promising paradigm for generalist robotic manipulation. A common design in current architectures maps language instructions and visual observations to actions in a single forward pass. While conceptually simple, this formulation entangles instruction comprehension, spatial scene understanding, and motor control within a single learning objective. As a result, the action expert must implicitly relearn cognitive and perceptual capabilities already present in the pretrained VLM, which can limit both learning efficiency and generalization. We introduce AVP (Action with Visual Primitives), an end-to-end architecture that implements this visual-primitive-centric interface: the VLM infers the next-stage target and emits visual-primitive tokens that condition a flow-matching action expert, with supervision derived from end-effector kinematics. Real-robot experiments on general pick-and-place tasks show that AVP improves the success rate by 27.61% over pi_0.5 and outperforms other recent methods, with consistent gains in data efficiency, spatial-compositional generalization, and object-level transfer.
Tsinghua AI