Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of detecting hallucinations in Arabic Islamic question answering by developing a two-step prediction system that identifies whether an answer generated by a large language model is hallucinated and selects the verified answer from closely related candidates. Utilizing the fine-tuned \texttt{google/gemma-4-12B-it} model, the authors achieved a Macro-F1 score of 0.928 for hallucination detection and an option accuracy of 0.895 for answer selection, resulting in a combined score of 0.912. The findings highlight the model's effectiveness in identifying hallucinations, although distinguishing verified answers from plausible alternatives remains a significant challenge.
Hallucination detection in Arabic Islamic Q&A systems can achieve over 92% accuracy, but selecting the correct answer from similar options proves to be a tougher nut to crack.
Large language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for \textsc{HalluScoring 2026} Task 2.1, \textit{Islamic Hallucination Detection and Find the Truth}. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge dataset provided by the shared task, which contains 600 question--answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned \texttt{google/gemma-4-12B-it} model and uses deterministic decoding during inference. The generated outputs are normalized to extract the hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.