Search papers, labs, and topics across Lattice.
This paper introduces Foresight, a novel framework that leverages pretrained Vision-Language Models (VLMs) to enhance open-world mapless navigation by iteratively refining motion plans based on environmental cues and language instructions. By alternating between proposing motion plans and critiquing them in the context of the language goal and visual environment, Foresight enables the model to adaptively focus on relevant cues that influence navigation. The approach achieves a 37% improvement in task success rates and a 52% reduction in intervention needs across various real-world environments compared to existing state-of-the-art methods.
Iterative reasoning with Vision-Language Models can drastically improve mapless navigation success rates by dynamically identifying and refining relevant environmental cues.
Open-world mapless navigation from sparse language instructions requires resolving underspecified goals and inferring which environmental cues are relevant for reaching the goal. For instance, reaching an out-of-view destination may require interpreting ramps, signs, or detours that reveal where to go or which route to take. Prior works are limited by their reliance on known navigation factors and closed-set factor categories, or identify cues before motion planning and miss plan-dependent cues. We argue that pretrained Vision-Language Models (VLMs) can discover novel instruction-relevant cues, but require adaptation to focus on which cues matter and how they should influence motion planning. We realize these ideas in Foresight, a test-time framework in which a finetuned VLM alternates between proposing image-space motion plans and critiquing them using the language goal and visual context. Subsequent plans are conditioned on prior critiques, enabling iterative motion refinement before execution. To align plan critiques and refinements with open-set behavior preferences, we learn a reward model from human feedback and use it to post-train the VLM with reinforcement learning in the plan-critique loop. In offline evaluations and 6 real-world environments, Foresight improves average task success by 37% and reduces interventions per mission by 52% relative to state-of-the-art test-time reasoning and foundation-model baselines, while running in real-time on a Jetson AGX Orin. We will release code, data, and training details to support future work on test-time reasoning for robot motion refinement. Additional videos at: https://amrl.cs.utexas.edu/foresight
