Search papers, labs, and topics across Lattice.
This paper introduces Grounding-Aware Sensitivity by Perturbation (GASP), a novel span-level detector designed to identify hallucinations in retrieval-augmented generation (RAG) outputs by measuring how much the likelihood of each answer sentence depends on its supporting evidence. By evaluating the log-likelihood drops and Jensen-Shannon divergences when context is altered, GASP effectively distinguishes between grounded and hallucinated sentences, achieving a response-level AUC of approximately 0.73 on the RAGTruth benchmark. The method outperforms traditional perplexity measures and other baselines, demonstrating its potential as a training-free detector that can operate without labeled data.
GASP reveals that grounded sentences are significantly more sensitive to context removal than hallucinated ones, providing a powerful new way to detect hallucinations in RAG outputs.
Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why. To close this gap, we introduce Grounding-Aware Sensitivity by Perturbation (GASP), a span-level detector that scores each answer sentence by how strongly its likelihood depends on the retrieved evidence, a quantity we term grounding sensitivity. GASP holds the answer fixed and re-scores it under the full context, under no context, and with each chunk removed, then measures the log-likelihood drops and Jensen-Shannon divergences (JSD). The likelihood of a grounded sentence collapses once its supporting passage is removed, whereas a hallucinated sentence is almost unaffected, a contrast we interpret by casting decoding as a random nonlinear iterated function system (RNIFS). We evaluate GASP on three benchmarks (RAGTruth, TofuEval, RAGBench) with three instruction-tuned scorers from two model families (Qwen2.5-0.5B, Qwen2.5-1.5B, and SmolLM2-1.7B) under a leakage-clean protocol. On RAGTruth it reaches a response-level area under the ROC curve (AUC) of about 0.73 and a span-level AUC of about 0.67, improving significantly over perplexity and by clear margins over length, whole-context natural language inference (NLI), and self-consistency baselines. The only baseline competitive at the span level is a well-configured chunk-level entailment verifier, which requires a separate model, whereas a training-free threshold on the grounding features matches the trained classifier without labeled data and serves as the default detector. Beyond RAGTruth, the signal transfers to TofuEval but not to short-answer question answering in RAGBench, showing GASP is best suited to outputs constructed from the retrieved context rather than answers recoverable from parametric knowledge.
