Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper systematically benchmarks five lightweight, CPU-based methods for hallucination detection in AI, focusing on their performance across question answering, dialogue, and summarisation tasks. The findings reveal that while no single method consistently outperforms the others, an ensemble approach excels in question answering, highlighting the task-dependent nature of detection efficacy. Notably, all methods struggle significantly with summarisation, indicating a critical gap in GPU-free hallucination detection capabilities.
Task-dependent performance reveals that lightweight hallucination detection methods can significantly vary, with ensembles outperforming individual models in specific scenarios.
Hallucination detection has become a pressing requirement for trustworthy AI deployment at scale. The most accurate detection methods depend on GPU-intensive inference, proprietary API calls, or white-box access to the generating model. This puts them out of reach for resource-constrained researchers and practitioners. In this paper, we explore a practical alternative: how well can hallucination detection perform using only lightweight, CPU-feasible methods built on publicly available models? We systematically benchmark five such methods: ROUGE-L, semantic similarity, BERTScore, a Natural Language Inference (NLI) detector based on a FEVER-trained DeBERTa model, and a score-level ensemble of similarity and NLI. We evaluate them across all three tasks of the HaluEval benchmark: question answering (QA), dialogue, and summarisation. We calibrate each method on a held-out validation split and evaluate it on 2,000 test instances per task. We find that no single method dominates and performance is highly task-dependent. The ensemble performs best on QA (F1 = 0.792, AUC-ROC = 0.873), the NLI detector leads on dialogue (AUC-ROC = 0.713), and all five methods degrade to near-random performance on summarisation (AUC-ROC between 0.469 and 0.574). This task-dependence and the systematic failure on summarisation map the practical frontier of GPU-free hallucination detection. They give practical guidance for method selection under computational constraints. All experiments run on a standard laptop CPU using public models.