Search papers, labs, and topics across Lattice.
This study investigates the cross-modal stability of multimodal foundation models by evaluating their ability to interpret semantically equivalent queries in both text and speech across English and Arabic languages. The authors introduce a novel benchmark featuring 10,150 culturally relevant images and assess models' performance through a contrastive triplet framework, revealing significant inconsistencies in judgments based on modality and language. Key findings indicate that speech input exacerbates partial reasoning failures, highlighting the limitations of current multimodal models in delivering reliable outputs across different contexts.
Multimodal models struggle with consistency, showing significant judgment discrepancies between text and speech inputs, especially in Arabic contexts.
Multimodal foundation models are increasingly used in speech-first assistants that must interpret spoken queries and produce visually grounded decisions. Yet it remains unclear whether semantically equivalent queries yield consistent judgments across modality (text vs. speech) and language (English vs. Arabic). We introduce a speech-augmented visually grounded contrastive triplet benchmark spanning 10,150 culturally grounded images from 18 MENA countries, where each image is paired with one supported statement and two plausible but unsupported alternatives. We define contrastive instability as the conditional rate at which a model fails to resolve all statements within a triplet, isolating fragmented reasoning from complete failure. Evaluating recent multimodal models under text and speech in English and Arabic, we find that modality and language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy, with speech amplifying partial failures. We make the benchmark publicly available to the community.
