Search papers, labs, and topics across Lattice.
This paper introduces the Modality Maturity Index (MMI), a comprehensive benchmark for evaluating the multimodal capabilities of large language models across five modalities: text, image, audio, video, and document. The MMI consists of 893 carefully crafted questions that assess both the presence and correctness of responses across multiple modalities, with results indicating low performance scores (MPS ranging from 15.6 to 34.9) for leading multimodal models. Additionally, the study validates the reliability of LLM judges against human annotators, achieving a 70.8% agreement rate, highlighting the potential for LLMs to assist in multimodal evaluation.
Multimodal models struggle significantly, with leading systems scoring as low as 15.6 on the new Modality Maturity Index, revealing critical gaps in their capabilities.
Frontier language models are increasingly marketed as omni systems that can perceive and respond across modalities. Existing evaluation frameworks, however, focus almost exclusively on bimodal understanding, typically text plus one other modality. We propose the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video and document) and combinations of up to three modalities in both inputs and outputs. MMI consists of 893 questions, each carefully crafted to require the model to demonstrate its understanding of multiple input modalities and to generate responses that incorporate various output formats. The questions are designed to be self-contained, with clear expectations for the correct modality or mix of modalities required for an accurate response. Every MMI prompt carries human-authored rubric criteria for each output modality expected in the response; a model's MMI Value expresses the average of the per-modality scores for each prompt. Because low scores can reflect either failure to generate a modality (lack of presence) or failure to generate correct content, we introduce also a supplementary Modality Presence Score (MPS), a per-prompt F1 over the expected output modalities. Applying MMI to five frontier multimodal models, we find that the MPS ranges from only 15.6 (Claude Opus 4.6) to 34.9 (GPT-5.4). Given the low availability of returned modalities to even grade, we report MPS as our main result pending model improvements. To assess the viability of judging output correctness with LLM judges and rubrics, we run a separate experiment with custom generation tools. On the assets that generates, we find that an LLM judge applying the rubrics agrees with rubric-blind human annotators (who score the outputs directly and never see the criteria) on 70.8% of judgments.
