Search papers, labs, and topics across Lattice.
This paper introduces ReSiReg, a feature reconstruction method designed to enhance spatial consistency in language-conditioned robotic tasks by leveraging spatially consistent intermediates from Vision-Language Models (VLMs). By clustering VLM intermediates into visual prototypes and reconstructing language embeddings as soft mixtures, the authors demonstrate significant improvements in dense language-grounded retrieval and spatially consistent target activations in real-world manipulation tasks. The approach not only yields better quantitative results on OVSS and 3D mapping but also offers a compact 25M dense VLM that remains competitive with larger baselines like ViT-B.
Spatially consistent language embeddings can drastically improve robotic task performance, making ReSiReg a game-changer for real-world applications.
Vision-Language Models (VLMs) enable robots to follow open-language instructions. However, dense VLM embeddings have shown to be noisy and lack spatial consistency. This is problematic for robotic applications, which require simultaneous reasoning over semantics and 3D space. We examine spatial structure across recent VLMs and propose ReSiReg, a feature reconstruction method that uses spatially consistent VLM intermediates to improve dense language-grounded retrieval. ReSiReg clusters intermediates into visual prototypes, derives their language descriptors, and reconstructs each patch as a soft mixture of prototype-level language embeddings. We evaluate quantitatively on OVSS and 3D mapping across backbones, and qualitatively in real-world manipulation scenes. Quantitative results show improved dense retrieval; manipulation scenes show more spatially consistent target activations. We further provide a compact 25M dense VLM for robotic applications, substantially smaller than and competitive with ViT-B baselines. Available at https://resireg.github.io
