Search papers, labs, and topics across Lattice.
This paper introduces NN-PPI, a novel calibration method that enhances the performance of small language models (SLMs) for claim check-worthiness detection without requiring retraining. By applying a lightweight post-hoc layer at inference time, NN-PPI achieves significant improvements in accuracy, with weighted F1 scores increasing by 12% to 33.80% across various baseline models. The approach not only elevates the performance of few-shot SLMs to levels comparable with larger LLMs but also enhances a fine-tuned model already in production, demonstrating its practical utility in cost-effective automated fact-checking systems.
NN-PPI enables small language models to rival the accuracy of larger models, making automated fact-checking both scalable and affordable.
Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challenge at an early-stage startup: running large language models (LLMs) over every incoming claim is cost- and latency-prohibitive, yet smaller models sacrifice accuracy. We propose NN-PPI, a pointwise extension of Prediction-Powered Inference (PPI) that calibrates model predictions at inference time as a lightweight post-hoc layer, without re-training the underlying model. NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model, bringing SLMs on par with larger LLMs. Beyond few-shot SLMs, NN-PPI further improves a production-deployed fine-tuned model, demonstrating that residual calibration is complementary to supervised fine-tuning. By recovering LLM-level accuracy from models that are an order of magnitude cheaper to serve, it makes accurate check-worthiness detection substantially cheaper to operate at scale. Our code and data can be found at https://anonymous.4open.science/r/arr-claim-worthiness-F237.
