Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SpanCalib-VLM, a hybrid dual-system designed to enhance hallucination detection in Large Vision-Language Models (LVLMs) by combining a multimodal sequence tagger with a fine-tuned generative VLM. The approach addresses the challenges of overconfidence and high inference latency in generative models while improving span localization and calibration through a Union-Calibrated Fusion strategy. The resulting model achieves a Pearson calibration correlation of 0.41 and an overall detection accuracy of 70.7% on the SHROOM-Visions English evaluation split, demonstrating significant improvements in both speed and accuracy.
A hybrid model that combines generative and discriminative approaches achieves a remarkable 70.7% detection accuracy while significantly improving calibration in hallucination detection for LVLMs.
Detecting hallucinations in Large Vision-Language Models (LVLMs) requires both accurate span localization and well-calibrated confidence scores. Fine-tuned generative VLMs excel at identifying hallucinated text spans but suffer from overconfidence and high inference latency. Discriminative sequence taggers offer deterministic speed and superior calibration but exhibit conservative span recall. We present SpanCalib-VLM, a hybrid dual-system for the SHROOM-Visions Shared Task that combines a multimodal sequence tagger, consisting of XLM-RoBERTa-Large fused with a SigLIP vision encoder via cross-attention, with our fine-tuned generative VLM (Qwen3.5-4B-SHROOM-SFT). Through a Union-Calibrated Fusion strategy, candidate spans from the generative model are re-scored with calibrated probabilities from the sequence tagger. On the SHROOM-Visions English evaluation split, our ensemble achieves a Pearson calibration correlation of 0.41 and an overall IoU of 0.39, with a clean-response IoU of 0.91} and overall detection accuracy of 70.7%. We make our model weights and code publicly available.