Search papers, labs, and topics across Lattice.
This paper introduces Traj-VLN, a method that fine-tunes vision-language models (VLMs) to learn navigation interactions in 2D pixel space through autoregressive trajectory generation, rather than relying on depth or 3D geometric information. By decomposing the vision-and-language navigation task into a sequence of sub-tasks, the model predicts pixel coordinates based on linguistic instructions and historical observations. The results show that this approach significantly enhances performance in vision-and-language navigation, achieving state-of-the-art results with limited computational resources and training data.
Fine-tuning vision-language models in pixel space leads to a dramatic boost in navigation performance, outperforming traditional methods reliant on 3D information.
Benefiting from the powerful priors embedded in large-scale pre-training data and the emerging commonsense reasoning ability, large language models (LLMs) have shown unprecedented generalization capabilities in many research fields. Recently, projecting visual embeddings into the language space via vision-language models (VLMs) to achieve sim-toreal and cross-scene generalization has become a prevailing paradigm in the field of Vision-and-Language Navigation in Continuous Environments (VLN-CE). VLN requires an embodied agent to navigate through unseen environments following natural linguistic instructions. We emphasize that a VLN task can be decomposed into a sequence of sub-tasks, each corresponding to a process of 3D spatial interaction with the environments described by instructions such as "walk to the end of the sofa and turn left." However, such spatial interactions involving moving into the image along the direction of depth sensing are puzzling for VLMs as they were predominantly trained on conversations with RGB images. Rather than incorporating depth or 3D geometric information-which VLMs rarely encounter during pretrainingwe propose an alternative approach: fine-tuning VLMs to learn navigation interactions directly in 2D pixel space through autoregressive trajectory generation. Given a linguistic instruction and historical observations, our model sequentially predicts a series of pixel coordinates, drawing a trajectory from the bottom center of the current observation. While prior work has proved that pixel-goal supervision outperforms learning of discrete actions, our experiments further verify that the supervision of pixel-space trajectory significantly enhances VLN performance. Moreover, we demonstrate that our flagship model achieves state-of-the-art level performance with relatively limited computational resources and training data.
