Search papers, labs, and topics across Lattice.
This paper introduces a new approach to grounded claim factuality checking that frames the task as a true/false reading comprehension problem and prompts LLMs with explicit human test-taking strategies. This method drastically reduces token usage (over 80%) while achieving competitive or state-of-the-art performance on factuality benchmarks compared to more expensive alternatives. The authors further demonstrate that smaller language models, fine-tuned with self-revision, can effectively replace LLMs in this pipeline, maintaining performance with significantly lower inference costs.
LLMs can check facts with 80% fewer tokens by mimicking human test-taking strategies, and surprisingly, smaller models can learn to do it just as well.
Grounded claim factuality checking is important for large language model (LLM) applications such as retrieval-augmented generation, as it helps users assess the correctness of generated outputs. Existing metrics using entailment classifiers require dataset-specific threshold tuning, while LLM-based approaches often use direct prompting, which underutilises the reasoning capabilities of LLMs. We address this by formulating grounded claim factuality checking as a true/false reading comprehension task and prompting LLMs with explicit test-taking strategies for efficient reasoning. Our method reduces token usage by over 80% compared to unguided open-ended reasoning, and achieves competitive performance to more expensive alternatives across two factuality benchmarks, setting a new state of the art on one. To further reduce inference cost, we train small language models (SLMs) to replace LLMs in the checking pipeline. Using supervised fine-tuning (SFT) and a self-revision mechanism, the SLMs learn to improve their factuality judgements. Experimental results show that the resulting SLMs perform on par with strong baselines, combining low inference costs with generating supporting rationales to support interpretability. Code and datasets will be released upon acceptance.
