Search papers, labs, and topics across Lattice.
HalluTruthQA-4K is a comprehensive dataset designed to enhance the detection of hallucinations and truth verification in Arabic language models, featuring 4,000 expert-curated question-answering instances across four domains. Unlike existing resources that provide binary labels, this corpus offers detailed annotations, including character-level erroneous spans and human-written explanations for hallucinated responses. The dataset's structured approach to error identification and verification aims to improve the reliability of Arabic LLMs and serves as the official dataset for the HalluScoring 2026 shared task.
HalluTruthQA-4K reveals that over 40% of Arabic model-generated responses contain hallucinations, with detailed annotations that pinpoint specific errors and their explanations.
Large language models can generate fluent Arabic answers while introducing factual errors that are difficult to identify and verify. Existing Arabic hallucination resources often assign a binary label to an entire response, indicating whether it is hallucinated or non-hallucinated, but provide limited information about the exact erroneous content, the reason for the error, or the correct factual answer. We present HalluTruthQA-4K, an expanded version of the HalluTruthQA resource containing 4,000 expert-curated Arabic question-answering instances across four knowledge-intensive domains: Islamic knowledge, history, science, and geography. Serving as the official dataset for Track 2 of the HalluScoring 2026 shared task, HalluTruthQA-4K extends our original corpus to 4,000 instances. Each instance pairs an Arabic question with a model-generated response, a verified reference answer, and five plausible distractors. Hallucinated responses are additionally annotated with character-level erroneous spans, human-written explanations, and hierarchical hallucination types. The corpus contains 1,643 hallucinated and 2,357 non-hallucinated responses, with 1,843 annotated erroneous spans. We describe the resource construction and annotation methodology, including question selection, controlled answer generation, candidate construction, expert annotation, independent verification, adjudication, and quality control. We also document the annotation guidelines, taxonomy, data format, inter-annotator agreement, and corpus statistics. HalluTruthQA-4K provides a reusable resource for hallucination detection, span-level error localization, explanation generation, factual verification, and the broader evaluation of factual reliability in Arabic language models.
