Search papers, labs, and topics across Lattice.
This paper introduces robot-centric pointmaps to address the frame mismatch in vision-language-action (VLA) models that arises when actions are defined in the robot's 3D coordinate frame while observations are made in the camera frame. By providing a 3D geometric representation of the scene in the robot's frame, pointmaps enable better generalization across diverse camera setups and improve action prediction. Experimental results on the RoboCasa dataset show that pointmaps significantly enhance the performance of both pi0.5 and SmolVLA models, particularly in scenarios with unseen camera placements during real-robot experiments.
Robot-centric pointmaps can drastically improve action prediction in VLA models, especially when faced with diverse camera viewpoints.
Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.
