Search papers, labs, and topics across Lattice.
This paper critiques the current evaluation methods for Vision-Language Models (VLMs) by highlighting the overlooked complexities of multi-script languages, specifically focusing on Punjabi. The authors introduce the PuMVR benchmark, consisting of 1,000 parallel image-text instances across three Punjabi scripts, revealing a significant Script Gap where VLMs demonstrate up to 16% accuracy differences across scripts. They propose the Script Consistency Rate (SCR) as a new metric to address these disparities, emphasizing that existing evaluations fail to capture the true multilingual capabilities of VLMs.
VLMs are failing to deliver equitable performance across different scripts, with accuracy gaps of up to 16% that could hinder access for billions of users.
Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script languages. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR.
