Search papers, labs, and topics across Lattice.
This paper introduces a decoupled framework for Vision-and-Language Navigation (VLN) that separates spatial state estimation from high-level semantic planning, using an interactive metric world representation to provide MLLMs with rich environmental information. The framework incorporates counterfactual reasoning to improve MLLM decision-making while ensuring physical action validity through the metric representation. Experiments demonstrate state-of-the-art zero-shot performance on R2R-CE and RxR-CE benchmarks, and successful zero-shot sim-to-real transfer to a TurtleBot 4 and a custom-built aerial drone.
Forget predefined maps: this VLN agent uses a learned, interactive world representation to achieve SOTA zero-shot performance and seamless sim-to-real transfer.
A navigable agent needs to understand both high-level semantic instructions and precise spatial perceptions. Building navigation agents centered on Multimodal Large Language Models (MLLMs) demonstrates a promising solution due to their powerful generalization ability. However, the current tightly coupled design dramatically limits system performance. In this work, we propose a decoupled design that separates low-level spatial state estimation from high-level semantic planning. Unlike previous methods that rely on predefined, oversimplified textual maps, we introduce an interactive metric world representation that maintains rich and consistent information, allowing MLLMs to interact with and reason on it for decision-making. Furthermore, counterfactual reasoning is introduced to further elicit MLLMs' capacity, while the metric world representation ensures the physical validity of the produced actions. We conduct comprehensive experiments in both simulated and real-world environments. Our method establishes a new zero-shot state-of-the-art, achieving 48.8\% Success Rate (SR) in R2R-CE and 42.2\% in RxR-CE benchmarks. Furthermore, to validate the versatility of our metric representation, we demonstrate zero-shot sim-to-real transfer across diverse embodiments, including a wheeled TurtleBot 4 and a custom-built aerial drone. These real-world deployments verify that our decoupled framework serves as a robust, domain-invariant interface for embodied Vision-and-Language navigation.
