Search papers, labs, and topics across Lattice.
This paper introduces SoftNav, a novel approach that injects entity-level 3D continuous representations into vision-language models (VLMs) for improved goal-directed embodied navigation. By directly embedding 3D scene tokens into the VLM's hidden space, SoftNav significantly reduces the representation gap observed with traditional text-based methods, achieving superior performance on the HM3D-OVON benchmark with a success rate of 74.2%. The method not only outperforms existing techniques but also enables zero-shot transfer to other navigation tasks and real-world applications without the need for retraining.
Directly injecting 3D scene tokens into VLMs boosts navigation success rates and enables seamless transfer to real-world scenarios without retraining.
In goal-directed embodied navigation, where an agent must locate a specified target in an unseen environment, 3D scene understanding and navigation reasoning must work in concert. Current approaches transmit 3D scene information to vision-language models (VLMs) through text, suggesting a representation gap in our tested configurations; a controlled ablation confirms that direct embedding-level transfer significantly outperforms the evaluated text serialization formats. We introduce SoftNav, which injects entity-level 3D continuous representations -- one token per detected object or frontier -- into a VLM's hidden space as soft tokens through a lightweight projector. With the 3D encoder and VLM frozen, only ~1,200 samples and ~17M trainable parameters are needed. On HM3D-OVON, SoftNav achieves 74.2%/68.3%/66.7% SR across three splits, surpassing all prior methods in both SR and SPL; the same navigation policy transfers zero-shot to GOAT-Bench (67.2% SR), SG3D (47.2% s-SR), and real-world robot deployment without retraining or architectural modification. Injecting 3D scene tokens directly into VLMs bridges the representation gap, enabling transferable navigation with minimal training.
