Search papers, labs, and topics across Lattice.
This paper addresses the limitations of existing Multimodal Large Language Models (MLLMs) in understanding sports videos by introducing SportMV-Bench, a new benchmark that evaluates multi-view video comprehension. The benchmark consists of 787 multi-view video bundles and 2,592 question-answer pairs, revealing that current MLLMs struggle with fine-grained visual perception and effective view selection. By implementing the SportMV-Agent framework, which facilitates active view selection and evidence-grounded reasoning, the authors achieve a notable 14.46% relative improvement over the best-performing MLLM baseline.
Current MLLMs miss critical insights from multi-view sports videos, but a new agentic framework boosts performance by over 14% through smarter view selection.
Recent Multimodal Large Language Models (MLLMs) achieve strong performance on single-view video understanding benchmarks. However, sports videos involve dense occlusion, rapid motion, and complex interactions that are difficult to resolve from a single viewpoint. In practice, sports events are recorded from multiple camera angles, providing complementary evidence used by referees. Yet, no existing benchmark evaluates MLLMs on multi-view sports video understanding. To address this gap, we introduce SportMV-Bench, a comprehensive benchmark built from official match recordings, through a dedicated pipeline combining LLM-based generation, MLLM-based verification, and human filtering to ensure quality and consistency. SportMV-Bench containing 787 multi-view video bundles and 2592 question-answer pairs across three categories: Perception-Aware Recognition (PAR), Rule-aware Event Interpretation (REI), and Adjudicative Decision Reasoning(ADR). Our analysis shows that current MLLMs fail to effectively exploit multi-view information, with the bottlenecks lying in fine-grained visual perception and view selection rather than logical reasoning or domain knowledge. We propose SportMV-Agent, an agentic framework that orchestrates an iterative loop of active view selection, perception tool execution, and evidence-grounded reasoning, achieving a significant 14.46% relative improvement over the strongest MLLM baseline.
Microsoft Research