Search papers, labs, and topics across Lattice.
This paper investigates Post Hallucination Reasoning (PHR) in vision language models (VLMs), revealing that hallucinated captions can enhance accuracy in vision language tasks while having inconsistent effects in text-only tasks. The authors introduce HIVE, an evaluation infrastructure that allows for controlled comparisons between faithful and hallucinated captions, uncovering structured patterns in how hallucinated semantics influence downstream predictions. The findings suggest that hallucinations, rather than being mere errors, can reshape reasoning dynamics and broaden semantic coverage, highlighting the need for a deeper understanding of this phenomenon in multimodal systems.
Hallucinated captions can paradoxically boost accuracy in vision language tasks, challenging the notion that they are purely detrimental.
Hallucinations in vision language models (VLMs) are commonly treated as semantic errors, yet they often arise from partial or ambiguous visual evidence. Prior work mainly focuses on detecting or suppressing hallucinations at generation time, leaving the subsequent reasoning stage largely unexplored. In this work, we study Post Hallucination Reasoning (PHR), the stage in which hallucinated semantics enter the model's inference context and influence downstream predictions. To systematically investigate PHR, we introduce HIVE, Hallucination Inference and Verification Engine, an evaluation infrastructure that enables controlled comparisons between faithful and hallucinated captions. Across nine tasks and nine models, we observe structured modality dependent patterns: hallucinated captions often improve accuracy on vision language tasks, while text only tasks exhibit limited or unstable effects. Further analyses show that hallucinated cues broaden semantic coverage and reshape reasoning dynamics while preserving stable inference. These findings highlight that hallucinated semantics may influence downstream reasoning once they enter the model's inference context. Understanding this post hallucination stage is important for improving the reliability and interpretability of multimodal reasoning systems. Code is publicly available at https://github.com/hefengcs/HIVE.
