Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces LaTraNav, a framework that learns language-conditioned traversability representations for adaptive visual navigation, addressing limitations of explicit costmaps and viewpoint-dependent segmentation. The asynchronous architecture pairs a slow vision-language model (VLM) for latent traversability and goal representation with a fast flow-matching planner, achieving improved planning performance and a $6.05\times$ increase in path-update rate. The system effectively segments traversability and localizes goals based on language instructions, enabling adaptive path planning that outperforms methods relying on explicit segmentation masks.
Language instructions directly shape latent traversability representations, enabling adaptive visual navigation that is $6\times$ faster than explicit segmentation-based approaches.
Traversability is essential for visual navigation but varies with robot capabilities and user preferences. Conventional pipelines often rely on explicit costmaps or segmentation masks with predefined criteria, requiring hand-crafted rules and careful tuning. Moreover, viewpoint-dependent segmentation masks complicate asynchronous planning under perception latency. We present LaTraNav, a framework that learns language-conditioned traversability representations for adaptive visual navigation. Its asynchronous architecture combines a slow vision-language model that produces latent representations of traversability and navigation goals, with a fast flow-matching planner conditioned on these representations. To train the system, we develop a simulation-based data generation pipeline with controllable trajectories, producing observations paired with language instructions, traversability maps, goal locations, and diverse trajectories. Photorealistic image translation further enhances visual realism. Evaluations on datasets from multiple sources demonstrate effective language-guided traversability segmentation and goal localization by the slow VLM, alongside adaptive pixel-space path planning by the fast planner. Latent conditioning improves planning performance over explicit segmentation masks, while asynchronous scheduling increases the path-update rate by $6.05\times$ at the same semantic-update rate.