Search papers, labs, and topics across Lattice.
The paper introduces VIGIL, a new benchmark dataset and framework for fine-grained categorization of hallucinations in image recontextualization by Large Multimodal Models (LMMs). VIGIL decomposes hallucinations into five categories: pasted object hallucinations, background hallucinations, object omission, positional & logical inconsistencies, and physical law violations. They propose a multi-stage detection pipeline leveraging an ensemble of open-source models to detect these hallucination types, demonstrating its effectiveness through experimental evaluations.
Hallucinations in multimodal models aren't a monolith: VIGIL breaks them down into five distinct categories, offering a much-needed fine-grained lens for evaluation.
We introduce VIGIL (Visual Inconsistency & Generative In-context Lucidity), the first benchmark dataset and framework providing a fine-grained categorization of hallucinations in the multimodal image recontextualization task for large multimodal models (LMMs). While existing research often treats hallucinations as a uniform issue, our work addresses a significant gap in multimodal evaluation by decomposing these errors into five categories: pasted object hallucinations, background hallucinations, object omission, positional & logical inconsistencies, and physical law violations. To address these complexities, we propose a multi-stage detection pipeline. Our architecture processes recontextualized images through a series of specialized steps targeting object-level fidelity, background consistency, and omission detection, leveraging a coordinated ensemble of open-source models, whose effectiveness is demonstrated through extensive experimental evaluations. Our approach enables a deeper understanding of where the models fail with an explanation; thus, we fill a gap in the field, as no prior methods offer such categorization and decomposition for this task. To promote transparency and further exploration, we openly release VIGIL, along with the detection pipeline and benchmark code, through our GitHub repository: https://github.com/mlubneuskaya/vigil and Data repository: https://huggingface.co/datasets/joannaww/VIGIL.
