Search papers, labs, and topics across Lattice.
VidHalLoc establishes a unified diagnostic benchmark to evaluate video hallucination detectors using 2,000 adversarial samples across VideoQA and captioning tasks spanning ontological and dynamic hallucination categories. Assessing detector reliability is critical as multimodal foundation models and video agents increasingly depend on automated verifiers for self-correction and alignment. Across fifteen evaluated methods, dedicated video hallucination detectors peaked at a remarkably low 34.63% overall accuracy, demonstrating that current detection systems fail to reliably verify spatiotemporal consistency.
Today's specialized video hallucination detectors peak at an abysmal 34.63% accuracy under adversarial conditions, revealing that the verifiers meant to guardrail multimodal foundation models are fundamentally untrustworthy.
Video-language models and video agents can produce hallucinations that conflict with spatiotemporal evidence. Existing benchmarks mainly evaluate model hallucinations, and heterogeneous mechanisms make detector reliability difficult to compare. We introduce VidHalLoc, a benchmark that evaluates hallucination detection methods under a unified diagnostic evaluation protocol using 2,000 adversarial hallucination samples across Video Question Answering and Video Captioning tasks, spanning Ontology and Dynamic hallucination categories. To construct VidHalLoc efficiently, we introduce VideoHALO, a Harness Engineering-informed multi-agent workflow that decomposes data construction into four executable stages supported by a memory system and a communication protocol. Evaluation of fifteen methods reveals that the four dedicated detectors peak at an Overall accuracy of only 34.63%, indicating limited reliability across video hallucination types [Dataset Repository: https://huggingface.co/datasets/wesfggfd/VidHalLoc].
