Search papers, labs, and topics across Lattice.
This paper introduces a Language Sensor Model (LSM) that effectively converts natural language spatial descriptions into calibrated spatial distributions, addressing the limitations of traditional mapping techniques and multimodal models in 3D environments. By integrating language observations with onboard perception through a probabilistic framework called VL-Map, the authors demonstrate that LSM can maintain calibrated covariance estimates, which is crucial for accurate spatial reasoning. The results show that LSM significantly improves target location predictions, achieving approximately 70% more probability mass on the true target compared to existing foundation models.
Language observations can be transformed into precise spatial distributions, enhancing robot navigation in complex environments.
Robots deployed in human-centric environments routinely receive natural-language descriptions of spatial information ("I left my backpack on the table") that reference parts of the world beyond their perceptual field of view. Traditional metric-semantic mapping ignores this signal, while off-the-shelf multimodal models remain limited in 3D spatial reasoning and are not directly amenable to fusion with other sensor modalities. To convert language observations into a calibrated spatial distribution, we train a Language Sensor Model (LSM) that maps each utterance and its scene-graph context to a multimodal distribution, with mixture weights encoding referential ambiguity (e.g., "which table") and component covariances encoding spatial uncertainty (e.g., where "on the table" the target lies). We then introduce VL-Map (Vision-Language Metric-Semantic Mapping), a probabilistic framework that treats these language predictions as stochastic observations and fuses them with onboard perception within a unified belief map. On the VLA-3D benchmark as well as on a real-world mobile robot, LSM is the only language predictor whose covariance estimates remain within the calibrated regime; fused into VL-Map, it leads to more accurate predictions of the target object location (~70% more probability mass on the true target compared to the strongest foundation-model baseline).
