Search papers, labs, and topics across Lattice.
This paper introduces Goal2Pixel, a novel approach to vision-language navigation that reformulates the task as navigable pixel grounding rather than low-level action prediction. By utilizing the image plane as a spatial interface, the model predicts visible navigable pixels, which are then converted into 3D waypoints for robot navigation, significantly reducing the number of required VLM inference calls. Goal2Pixel achieves state-of-the-art performance on R2R-CE Val-Unseen with a success rate of 54.1% while using only 7.75 VLM calls per episode, demonstrating a substantial efficiency improvement over traditional methods.
Navigating with fewer than 8 VLM calls per episode, Goal2Pixel redefines efficiency in vision-language navigation tasks.
Vision-language models (VLMs) have become a common foundation for vision-and-language navigation in continuous environments (VLN-CE). Yet most VLM-based methods cast navigation as low-level action prediction, an interface that is ambiguous, tied to short-horizon motion primitives, and inefficient due to repeated VLM querying. We propose Goal2Pixel, a pure pixel-based paradigm that reformulates VLN-CE as navigable pixel grounding. Rather than predicting actions, Goal2Pixel uses the image plane as a unified spatial interface between VLM reasoning and robot motion: the model predicts a visible navigable pixel to the agent, which is back-projected into a 3D waypoint for forward navigation. For non-forward actions, we append auxiliary directive regions to the image plane, where the left/right/bottom regions are interpreted as turning left, turning right, and stopping, respectively. To enable long-horizon navigation, we propose a visibility-aware keyframe memory for compact and informative history representation. To adapt pretrained VLMs to navigable pixel grounding, we introduce semantic embeddings and coordinate-aware auxiliary losses. Goal2Pixel achieves competitive state-of-the-art performance while requiring fewer VLM inference calls than prior methods. On R2R-CE Val-Unseen it achieves 54.1% SR and 52.5% SPL with just 7.75 VLM calls per episode, 6x fewer than the 46.62 required by direct action prediction at 32.9% SR. The same trend holds on RxR-CE.Project Page: https://baobao0926.github.io/Goal2Pixel/.
CMU ML