Search papers, labs, and topics across Lattice.
The paper introduces MMOU, a new benchmark for evaluating multimodal large language models (MLLMs) on long and complex real-world videos, requiring joint reasoning over visual, audio, and textual signals. MMOU comprises 15,000 questions paired with 9038 diverse videos, covering 13 skill categories that demand cross-modal and temporal integration. Evaluation of 20+ state-of-the-art models on MMOU reveals significant performance gaps, with the best closed-source model achieving only 64.2% accuracy and the strongest open-source model reaching 46.8%, highlighting the challenges of long-form omni-modal understanding.
Current multimodal models are surprisingly bad at understanding long, complex videos, struggling to integrate audio, visual, and text cues even for basic reasoning tasks.
Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over omni-modal (visual, audio, and textual) signals in long and complex videos remains largely unexplored. We introduce MMOU, a new benchmark designed to systematically evaluate multimodal understanding and reasoning under these challenging, real-world conditions. MMOU consists of 15,000 carefully curated questions paired with 9038 web-collected videos of varying length, spanning diverse domains and exhibiting rich, tightly coupled audio-visual content. The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time. All questions are manually annotated across multiple turns by professional annotators, ensuring high quality and reasoning fidelity. We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU. The results expose substantial performance gaps: the best closed-source model achieves only 64.2% accuracy, while the strongest open-source model reaches just 46.8%. Our results highlight the challenges of long-form omni-modal understanding, revealing that current models frequently fail to apply even fundamental skills in long videos. Through detailed analysis, we further identify systematic failure modes and provide insights into where and why current models break.
NVIDIA