Search papers, labs, and topics across Lattice.
This paper introduces Video-IFBench, a novel benchmark designed to evaluate the instruction-following capabilities of Multimodal Large Language Models (MLLMs) in video understanding contexts. By developing a comprehensive instruction taxonomy and a semi-automatic data construction pipeline, the authors generated 1.5K samples that assess models against diverse user-specified constraints. The evaluation reveals that current MLLMs struggle significantly with complex instructions, particularly those involving multiple constraints or conditional structures, highlighting a critical gap in their performance.
Current MLLMs falter in following complex video instructions, revealing a significant oversight in their evaluation metrics.
Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains under-explored. Real-world video understanding requires models not only to interpret video content correctly, but also to satisfy diverse user-specified constraints. Existing benchmarks focus primarily on task accuracy rather than instruction adherence, leaving this capability insufficiently evaluated. To address this gap, we introduce Video-IFBench, a comprehensive benchmark for evaluating instruction following in video understanding, where models must satisfy diverse user-specified constraints, including those grounded in visual and audio content. We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions, covering 32 task types and 39 manually designed constraint categories spanning both semantic and format requirements. To reduce annotation cost, we build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples. We conduct a large-scale evaluation of more than 20 recent MLLMs and show that video instruction following remains challenging for current models, especially for instructions with many constraints, semantic constraints, or complex conditional structures that require selecting the correct branch or path based on video content. We hope our work will facilitate future research on instruction following in video understanding scenarios.
