Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
OmniCapBench is introduced as a deep-structured evaluation framework for audio-visual captioning, addressing limitations of existing benchmarks by reframing the prediction target to atomic, verifiable units (entity references, visual shots, audio events) for fine-grained diagnostics. This approach enables reliable scoring through deterministic constraint checks and localized LLM-based semantic comparisons, effectively exposing MLLM perception errors like temporal grounding failures and cross-modal misalignment. Evaluation of frontier MLLMs on OmniCapBench reveals strong local perception but significant weaknesses in long-horizon audio-visual reasoning.
Frontier MLLMs exhibit strong local perception but falter on long-horizon audio-visual reasoning, with a single benchmark exposing specific failure modes like identity drift and cross-modal misalignment.
Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.