Search papers, labs, and topics across Lattice.
This paper evaluates VLMs' robustness to geometric transformations like rotations and scaling, revealing a significant performance drop even with simple transformations. The study spans diverse visual domains, including sketches, photographs, and abstract art, showing that VLMs struggle when semantic content is sparse. The authors find this fragility is consistent across architectures, model sizes, and prompting strategies, highlighting a gap between semantic understanding and spatial reasoning.
VLMs, despite excelling at semantic tasks, are surprisingly brittle when faced with basic geometric transformations like rotations and scaling.
This work investigates the fundamental fragility of state-of-the-art Vision-Language Models (VLMs) under basic geometric transformations. While modern VLMs excel at semantic tasks such as recognizing objects in canonical orientations and describing complex scenes, they exhibit systematic failures at a more fundamental level: lack of robust spatial invariance and equivariance required to reliably determine object identity under simple rotations, scaling, and identity transformations. We demonstrate this limitation through a systematic evaluation across diverse visual domains, including symbolic sketches, natural photographs, and abstract art. Performance drops sharply as semantic content becomes sparse, and this behavior is observed across architectures, model capacities, and prompting strategies. Overall, our results reveal a systematic gap between semantic understanding and spatial reasoning in current VLMs, highlighting the need for stronger geometric grounding in future multimodal systems.
