Search papers, labs, and topics across Lattice.
The paper introduces SpaAct, a training framework for adapting VLMs to Vision-Language Navigation (VLN) by enhancing spatial awareness through backward action reasoning and forward transition prediction. SpaAct employs two spatial activation tasks: Action Retrospection (inferring executed actions from visual transitions) and Future Frame Selection (predicting visual transitions conditioned on history and action). A Tri-factor Progressive Adaptive curriculum learning method (TriPA) is used to stabilize training, leading to state-of-the-art performance on VLN-CE benchmarks.
VLMs can achieve state-of-the-art Vision-Language Navigation performance by explicitly training them to reason about past actions and predict future visual transitions.
Vision-and-Language Navigation (VLN) aims to enable an embodied agent to follow natural-language instructions and navigate to a target location in unseen 3D environments. We argue that adapting VLMs to VLN requires endowing them with two complementary capabilities for acquiring such awareness, namely backward action reasoning (why) and forward transition prediction~(how). Based on this insight, we propose SpaAct, a simple yet effective training framework that activates the dynamic spatial awareness in VLMs. Specifically, SpaAct introduces two spatial activation tasks: Action Retrospection, which asks the model to infer the executed action sequence from visual transitions, and Future Frame Selection, which forces the model to predict the visual transitions conditioned on history and action. These two objectives provide lightweight supervision on both backward action reasoning and forward transition prediction, encouraging the model to build dynamic spatial awareness in a VLM-friendly way. To further stabilize adaptation, we design TriPA, a Tri-factor Progressive Adaptive curriculum learning method that organizes training samples from easy to hard, allowing the model to gradually acquire navigation skills from basic locomotion to long-horizon reasoning. Experiments on standard VLN-CE benchmarks show that SpaAct consistently improves VLM-based navigation and achieves state-of-the-art performance. We will release the code and models to support future research.
