Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to token-level hallucination detection by treating hallucinations as temporally extended spans rather than isolated tokens. By employing a sequence labeling method that integrates a 33-dimensional feature stream鈥攊ncluding text statistics, NLI entailment, and language model surprisal鈥攗sing a Bidirectional Gated Recurrent Unit (BiGRU), the authors achieve an AUC of 0.840, significantly outperforming a logistic-regression baseline by 11 points. The findings suggest that the temporal order of evidence is crucial for detection accuracy, with the model's performance being limited more by the feature set than by the architecture itself.
Detecting hallucinations in language models is significantly improved by considering the temporal context of tokens, achieving an AUC of 0.840 without relying on model internals.
Token-level hallucination detectors score each token independently from a single signal, and fail exactly when the generating model is confidently wrong. This paper instead treats hallucination as a temporally extended span and detects it by sequence labeling: each token is scored from a 33-dimensional feature stream that fuses text statistics, Natural Language Inference (NLI) entailment, and language model surprisal, with no access to model internals. A Bidirectional Gated Recurrent Unit (BiGRU) over these features reaches an AUC of 0.840 on RAGTruth (10 seeds), an 11-point gain over an independent logistic-regression baseline (p = 0.002, Wilcoxon signed-rank). A controlled decomposition attributes most of the gain to temporal order rather than model capacity: evidence propagates from confident positions to ambiguous neighbors within a span. The same 0.845 ceiling recurs across recurrent, state-space (Mamba), and attention architectures, locating the bottleneck in the feature set rather than the model. Because it reads only the generated text and external signals, the detector works on closed-source models, and it keeps working on text produced by language models it never saw during training, losing under 4% AUC.
