Search papers, labs, and topics across Lattice.
This study audits the order sensitivity of 18 multimodal large language models (MLLMs) using a novel framework called Facet-Probe, which evaluates models across five facets of order sensitivity. The findings reveal that none of the models are order-invariant, with flip rates varying between 24% and 50%, indicating significant sensitivity to input ordering. Additionally, while higher model capability correlates with fewer flips, the best-performing model still exhibits a 13.4% flip rate, suggesting that current prompt-level mitigation strategies are insufficient for achieving robust order invariance.
None of the 18 multimodal large language models audited are order-invariant, with flip rates revealing a staggering sensitivity to input ordering that challenges current evaluation practices.
Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.
Stanford HAI