Search papers, labs, and topics across Lattice.
This study investigates the impact of quantizing key-value caches in retrieval-augmented generation systems on the faithfulness of generated responses. By evaluating the Qwen2.5-7B-Instruct model under INT8 and INT4 quantization across RGB and HotpotQA datasets, the authors reveal that while INT8 maintains near-lossless accuracy and faithfulness, INT4 quantization significantly harms faithfulness, with over 90% of changes being negative despite maintaining factual correctness. These findings highlight the critical need for auditing faithfulness when deploying compressed caches, especially in scenarios with noisy retrieval.
Over 90% of faithfulness changes are negative under INT4 quantization, revealing a hidden risk in compressed retrieval-augmented generation systems.
Retrieval-augmented generation systems can precompute and store key-value caches of retrieved documents to avoid re-encoding context at every query. Quantizing these caches further reduces storage, but no prior work asks whether compression damages faithfulness, whether responses remain grounded in the retrieved evidence. Faithfulness and accuracy are not equivalent: a model can produce a correct answer that is no longer supported by the context it was given. We evaluate Qwen2.5-7B-Instruct under INT8 and INT4 quantization on RGB and HotpotQA, measuring both accuracy and faithfulness with a hallucination detector, NLI entailment, and an LLM judge. INT8 is near-lossless across both metrics. INT4 reduces accuracy and, more critically, even among answers that remain factually correct, over 90% of faithfulness changes are negative, i.e., accuracy metrics are blind to this regression. The harm grows under noisy retrieval and with more retrieved chunks. Faithfulness must be audited before compressed caches are deployed.
