Search papers, labs, and topics across Lattice.
This study investigates the sensitivity of vision-language models (VLMs) to the order of input presentation, revealing that image-first prompting consistently outperforms question-first prompting across multiple models and benchmarks. To address this modality order failure, the authors propose a test-time training method that not only closes the performance gap but also enhances the stronger image-first branch, fostering mutual consistency between both prompting orders. The research identifies a circuit-level failure related to modality order sensitivity and demonstrates that targeted adaptation can effectively realign representations, leading to improved model performance.
Vision-language models exhibit a surprising sensitivity to input order, with image-first prompting outperforming question-first by a significant margin, revealing a critical circuit-level failure.
We find that vision-language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and three benchmarks, image first prompting consistently outperforms question-first prompting, revealing a repeatable modality order failure. We use this gap to design an order-consistent test-time training method. Our method substantially closes the modality-order gap across all evaluated settings. Surprisingly, it also yields consistent improvements in the stronger image-first branch over the baseline, hence bootstrapping both orderings toward mutual consistency. Activation patching localizes the ordering failure to a narrow mid-network region where representations diverge sharply between prompt orders. We find that the test-time training method repairs this misalignment across layers. Together, our results identify modality-order sensitivity as a circuit-level failure in VLMs and demonstrate that simple, asymmetric test-time adaptation can effectively mitigate it and even improve performance over the baseline.
