Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces GAVEL, a novel task designed to tackle the challenges of hallucination and misalignment in vision-language models (VLMs) by jointly verifying, explaining, and localizing errors in image-text pairs. The authors present a new dataset and benchmark to facilitate systematic evaluation, demonstrating that existing strong models often fail on GAVEL, while a supervised baseline trained on human-annotated data shows significant improvements in grounding and explanation metrics. These findings highlight the critical need for better alignment between visual and textual information in VLMs, underscoring the potential of GAVEL as a framework for enhancing model reliability.
Even state-of-the-art vision-language models falter on GAVEL, revealing a significant gap in their ability to align text and images accurately.
Vision-language models (VLMs) often produce hallucinated or inconsistent outputs, where text and images are not properly aligned. Addressing this issue requires not only detecting misalignment but also explaining the discrepancy and localizing its visual evidence. We introduce GAVEL (Grounded Caption Error Verification and Localization), a task that jointly addresses verification, explanation, and localization for image-text pairs. To support systematic evaluation, we also present a corresponding dataset and benchmark. We further train a supervised baseline on the human-annotated training split to assess whether GAVEL provides learnable supervision for these abilities. Experiments show that even strong closed-source models struggle on GAVEL, while the supervised baseline yields consistent improvements across grounding and explanation metrics.