Search papers, labs, and topics across Lattice.
The paper introduces DualFact, a dual-layer multimodal framework for evaluating the factual correctness of procedural video captions by assessing both conceptual and contextual facts. They augment existing datasets with implicit argument annotations and contrastive fact sets to improve evaluation completeness and consistency. Experiments on YouCook3-Fact and CraftBench-Fact reveal that current models generate fluent but factually incomplete captions, and that video-grounded verification exposes more hallucinations than caption-only evaluation.
Multimodal language models are fluent liars: they produce convincing procedural video captions that are often factually incomplete, with systematic omissions and role-level inconsistencies exposed by video-grounded verification.
We introduce DualFact, a dual-layer, multimodal factuality evaluation framework for procedural video captioning. DualFact separates factual correctness into conceptual facts, capturing abstract semantic roles (e.g., Action, Ingredient, Tool, Location), and contextual facts, capturing their grounded predicate-argument realizations in video. To support complete and role-consistent evaluation, DualFact incorporates implicit argument augmentation (VIA) and contrastive fact sets. We instantiate DualFact in two modes: DualFact-T, which verifies facts against textual evidence, and DualFact-V, which verifies facts against video-grounded visual evidence. Experiments on YouCook3-Fact and CraftBench-Fact show that state-of-the-art multimodal language models produce fluent but often factually incomplete captions, with systematic omissions and role-level inconsistencies. DualFact correlates more strongly with human factuality judgments than standard metrics, particularly for contextual facts, and reveals that caption-only evaluation overestimates hallucinations compared to video-grounded verification. Overall, DualFact offers an interpretable and human-aligned evaluation protocol that highlights persistent challenges in multimodal factual grounding, extending beyond surface-level fluency.
