Search papers, labs, and topics across Lattice.
This paper introduces multi-view relational distillation (MVRD), a novel approach that enhances the geometric robustness of vision-language models (VLMs) by distilling patch-wise cosine similarities rather than directly matching teacher features. This method addresses the limitations of existing geometry grounding techniques, which either risk spurious representations or increase model size significantly. MVRD achieves superior performance in visual-spatial reasoning tasks while maintaining a lightweight model architecture, demonstrating its effectiveness across various 3D scene understanding applications.
Distilling geometric relationships rather than features allows VLMs to improve spatial reasoning without bloating model size or sacrificing language alignment.
Vision-language models (VLMs) have achieved strong image and video understanding, yet their visual-spatial representations remain geometrically fragile, leading to failures in spatial reasoning needed for embodied AI, robotics, and autonomous driving. Prior approaches to geometry grounding either fine-tune VLMs on spatial question answering, which can perpetuate spurious visual representations, or fuse features from large geometry-grounded vision models, which substantially increases model size at inference. Knowledge distillation from geometry-grounded vision models offers an alternative, but directly matching multi-view teacher features can disrupt the pretrained alignment between visual and textual representations, degrading object- and language-semantic capabilities. We propose multi-view relational distillation (MVRD), which distills patch-wise cosine similarities across views instead of the teacher features themselves. These relations encode geometric correspondences adequate for spatial understanding, while leaving the student representation underdetermined, allowing it to remain close to its pretrained vision- language space. Across representative VLMs, MVRD improves visual-spatial reasoning, outperforming supervised fine-tuning and feature distillation while approaching feature fusion methods with considerably fewer added parameters and lower latency. We show that MVRD makes visual representations more geometric while retaining language alignment, and generalizes to 3D scene understanding tasks such as object grounding, dense captioning, and question answering.
