Search papers, labs, and topics across Lattice.
RADIO-ViPE is introduced as an online semantic SLAM system that performs geometry-aware open-vocabulary grounding from monocular RGB video, associating natural language queries with 3D regions. It tightly couples multi-modal (vision and language) embeddings from foundation models with geometric scene information within an optimization framework using adaptive robust kernels to handle dynamic elements. Experiments on the TUM-RGBD benchmark show state-of-the-art results in dynamic environments compared to existing offline open-vocabulary methods.
Semantic SLAM can now understand free-form language queries and ground them in 3D space using only a monocular video feed, opening the door to robots that truly understand and interact with the world around them.
We present RADIO-ViPE (Reduce All Domains Into One -- Video Pose Engine), an online semantic SLAM system that enables geometry-aware open-vocabulary grounding, associating arbitrary natural language queries with localized 3D regions and objects in dynamic environments. Unlike existing approaches that require calibrated, posed RGB-D input, RADIO-ViPE operates directly on raw monocular RGB video streams, requiring no prior camera intrinsics, depth sensors, or pose initialization. The system tightly couples multi-modal embeddings -- spanning vision and language -- derived from agglomerative foundation models (e.g., RADIO) with geometric scene information. This coupling takes place in initialization, optimization and factor graph connections to improve the consistency of the map from multiple modalities. The optimization is wrapped within adaptive robust kernels, designed to handle both actively moving objects and agent-displaced scene elements (e.g., furniture rearranged during ego-centric session). Experiments demonstrate that RADIO-ViPE achieves state-of-the-art results on the dynamic TUM-RGBD benchmark while maintaining competitive performance against offline open-vocabulary methods that rely on calibrated data and static scene assumptions. RADIO-ViPE bridges a critical gap in real-world deployment, enabling robust open-vocabulary semantic grounding for autonomous robotics and unconstrained in-the-wild video streams. Project page: https://be2rlab.github.io/radio_vipe
