Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To resolve the severe inference latency bottleneck of querying autoregressive MLLMs at every timestep, NavJev reformulates zero-shot continuous Vision-and-Language Navigation (VLN) into modular visual compression followed by lightweight action selection. The architecture employs Action-Centric Visual Compression (ACVC) to fuse waypoint geometry, BLIP captions, and RAM semantic tags into compact candidate representations, while a Discriminative Action-Semantic Memory (DASM) filters redundant observations across trajectory steps. Evaluated on R2R-CE, NavJev matches competitive zero-shot agents with a 27.0% Success Rate and 22.4% SPL while executing at an ultra-low latency of just 0.65 seconds per navigation step.
Real-time embodied navigation no longer requires slow autoregressive MLLM rollouts at every timestep: compressing candidate waypoints into discriminative semantic tokens enables sub-second decisions on continuous navigation benchmarks with zero loss in task success.
Recent zero-shot Vision-and-Language Navigation (VLN) methods increasingly rely on multimodal large language models (MLLMs) to reason over visual observations, navigation instructions, and candidate actions. Although effective, repeatedly invoking autoregressive multimodal reasoning at every navigation step introduces substantial inference latency, limiting the responsiveness of embodied agents. We propose NavJev, an efficient VLN framework that reformulates online navigation from repeated multimodal generation into compact visual compression followed by lightweight typed action selection. Specifically, Action-Centric Visual Compression (ACVC) integrates waypoint geometry, BLIP captions, and RAM semantic tags into compact representations of candidate actions, while Discriminative Action-Semantic Memory (DASM) filters shared semantics and maintains discriminative action-specific evidence across navigation steps. Based on these representations, Jev directly performs structured probabilistic decisions over the available action set. Experiments on R2R-CE show that NavJev achieves 27.0% SR and 22.4% SPL with only 0.65 s per navigation step, while substantially reducing inference latency and cost compared with MLLM-based VLN methods. The project page is available at https://kai-sheng-caesar.github.io/NavJev/.