Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To resolve the limitations of lexical reference-matching in multimodal evaluation, the authors introduce CapQuiz, a reference-free benchmark that assesses video caption quality through information fidelity rather than n-gram overlap. By quantifying how effectively a generated caption enables answering human-verified, fine-grained multiple-choice questions, the framework evaluates both salient visual coverage and factual accuracy without being penalized by the one-to-many nature of video descriptions. The benchmark spans 10 descriptive and inferential question types across 24 domains, achieving significantly stronger correlation with human judgment than existing metrics while isolating specific model failure modes.
Standard video caption metrics penalize valid visual descriptions for mere lexical divergence, a failure mode solved by testing whether captions contain sufficient factual signal to answer downstream multiple-choice video comprehension questions.
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the ``one-to-many''nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality through the lens of information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.