Search papers, labs, and topics across Lattice.
This paper introduces AgenticNav, a novel zero-shot vision-and-language navigation harness that enhances the action space of vision-language models (VLMs) by allowing direct pixel selection and on-demand depth querying. By rethinking memory management through a compact map image and selective recall, AgenticNav mitigates the limitations of traditional waypoint predictors and irrelevant context accumulation. The approach achieves state-of-the-art performance on the R2R-CE benchmark, demonstrating superior zero-shot generalization and improved navigation capabilities in continuous environments.
AgenticNav revolutionizes zero-shot navigation by enabling VLMs to directly select target pixels and query depth, significantly expanding their operational capabilities.
Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs). However, existing methods typically rely on learned waypoint predictors to propose navigable actions. This severely limits the model's action space and fails to leverage depth inputs effectively. Moreover, memory is commonly handled by accumulating long textual or visual histories with substantial irrelevant context, or by retrieving cross-episode experiences, which weakens the zero-shot setting. In this paper, we rethink zero-shot VLN-CE as an agentic interface between the VLM and the environment, and present AgenticNav, a lightweight navigation harness that exposes action, depth, and memory as callable tools. Instead of choosing from predicted waypoints, the action tool allows the VLM to directly select a target pixel in RGB observations, converting it into executable motion. Depth is exposed through an on-demand pixel-depth tool, enabling the VLM to request precise metric distances only where they matter. For memory, AgenticNav provides a compact map image summarizing the historical trajectory, paired with a recall tool that allows the VLM to selectively revisit past visual observations without overwhelming the prompt context. On the R2R-CE benchmark, AgenticNav establishes new state-of-the-art (SOTA) performance among zero-shot methods given the same VLM backbone. Real-world validation further highlights its zero-shot generalization compared to prior methods. Ablations show that our action tool design outperforms traditional waypoint predictors, and that depth tool and agentic memory further contribute to navigation performance.
