Search papers, labs, and topics across Lattice.
This paper introduces LookStep, an efficient framework for Vision-Language Navigation (VLN) that integrates Language Centric Future State Modeling with Event Driven Rolling Memory to enhance navigation performance while reducing data and computational requirements. By leveraging language labels to predict future states and selectively recording observations, LookStep achieves a 49.7% success rate on the R2R-CE Val-Unseen dataset, outperforming existing methods with lower memory overhead. This approach addresses the limitations of traditional next-step action prediction paradigms, which often necessitate extensive training data and computational resources.
LookStep achieves a 49.7% success rate in VLN tasks while using significantly less data and memory than traditional methods.
Vision-Language Navigation (VLN) requires an embodied agent to follow natural-language instructions in unseen environments. Recent progress has been largely driven by Multimodal Large Language Models (MLLMs). Existing methods follow a next-step action prediction paradigm, supervising only the expert action, which requires a high quantity of data for training. They also rely on cognitive maps, accumulated historical frames, or external 3D tools to maintain states, leading to high computational and memory overhead. To realize resource efficiency VLN, we propose LookStep, a unified end-to-end framework that combines Language Centric Future State Modeling and Event Driven Rolling Memory that uses language labels to generate coarse-grained navigation progress and future states for each candidate action, while autonomously deciding whether to write each observation into a bounded rolling memory with a semantic role. We validate LookStep empirically. On VLN-CE tasks, LookStep outperforms existing methods under the same training settings, achieving a 49.7\% success rate on R2R-CE Val-Unseen with better memory efficiency and less data usage. Code and model is available at https://github.com/kunyang-YU/LookStep.
