Search papers, labs, and topics across Lattice.
This paper investigates the robustness of conformal factuality filtering for RAG-based LLMs, focusing on the trade-off between factuality and informativeness. The authors introduce new informativeness-aware metrics and evaluate conformal filtering across various benchmarks and model families. Their findings reveal that conformal filtering suffers from low usefulness at high factuality levels and is not robust to distribution shifts, while lightweight entailment-based verifiers can match or outperform LLM-based scorers with significantly less computation.
Conformal factuality for RAG breaks down when faced with distribution shifts or distractors, forcing a trade-off between factuality and informativeness.
Large language models (LLMs) frequently hallucinate, limiting their reliability in knowledge-intensive applications. Retrieval-augmented generation (RAG) and conformal factuality have emerged as potential ways to address this limitation. While RAG aims to ground responses in retrieved evidence, it provides no statistical guarantee that the final output is correct. Conformal factuality filtering offers distribution-free statistical reliability by scoring and filtering atomic claims using a threshold calibrated on held-out data, however, the informativeness of the final output is not guaranteed. We systematically analyze the reliability and usefulness of conformal factuality for RAG-based LLMs across generation, scoring, calibration, robustness, and efficiency. We propose novel informativeness-aware metrics that better reflect task utility under conformal filtering. Across three benchmarks and multiple model families, we find that (i) conformal filtering suffers from low usefulness at high factuality levels due to vacuous outputs, (ii) conformal factuality guarantee is not robust to distribution shifts and distractors, highlighting the limitation that requires calibration data to closely match deployment conditions, and (iii) lightweight entailment-based verifiers match or outperform LLM-based model confidence scorers while requiring over $100\times$ fewer FLOPs. Overall, our results expose factuality-informativeness trade-offs and fragility of conformal filtering framework under distribution shifts and distractors, highlighting the need for new approaches for reliability with robustness and usefulness as key metrics, and provide actionable guidance for building RAG pipelines that are both reliable and computationally efficient.
