Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
SOL-Nav converts egocentric RGB-D images into structured language descriptions composed of semantic, color, and depth information extracted from a grid-based representation. This structured language, combined with navigation instructions, is then fed into a pre-trained language model (PLM) for navigation. SOL-Nav achieves strong generalization on VLN benchmarks (R2R, RxR) and real-world deployments while significantly reducing model size and training data dependency by leveraging the PLM's reasoning capabilities.
Ditch the pixels: SOL-Nav proves you can achieve strong vision-language navigation by translating raw visual inputs into structured language, unlocking efficient generalization with pre-trained language models.
Vision-Language Navigation (VLN) requires an embodied agent to navigate complex environments by following natural language instructions, which typically demands tight fusion of visual and language modalities. Existing VLN methods often convert raw images into visual tokens or implicit features, requiring large-scale visual pre-training and suffering from poor generalization under environmental variations (e.g., lighting, texture). To address these issues, we propose SOL-Nav (Structured Observation Language for Navigation), a novel framework that translates egocentric visual observations into compact structured language descriptions for efficient and generalizable navigation. Specifically, we divide RGB-D images into a N*N grid, extract representative semantic, color, and depth information for each grid cell to form structured text, and concatenate this with the language instruction as pure language input to a pre-trained language model (PLM). Experimental results on standard VLN benchmarks (R2R, RxR) and real-world deployments demonstrate that SOL-Nav significantly reduces the model size and training data dependency, fully leverages the reasoning and representation capabilities of PLMs, and achieves strong generalization to unseen environments.