Search papers, labs, and topics across Lattice.
This paper introduces a refined security principle for Retrieval-Augmented Generation (RAG) systems, allowing only agents capable of System 2 reasoning to access untrusted documents, thereby mitigating knowledge-poisoning vulnerabilities. The authors develop novel metrics to measure the gap between misinformation detection and its downstream influence, empirically demonstrating that reasoning-capable models outperform standard models in robustness against corrupted evidence. This approach not only enhances security without the computational overhead of strict isolation but also lays a practical foundation for future RAG system designs.
Reasoning-capable models can significantly reduce the impact of misinformation in RAG systems without the heavy computational costs of isolation.
Retrieval-Augmented Generation (RAG) has significantly enhanced the performance of large language models (LLMs), yet these systems remain vulnerable to knowledge-poisoning attacks, in which misinformation in retrieved documents can influence the model's final outputs. Notably, an LLM may correctly detect that a document contains incorrect information while nevertheless being influenced by it. Prior work has addressed this vulnerability through the Cordon Principle, which prevents models responsible for final answer synthesis from directly accessing raw evidence. Although effective, this strict isolation can introduce substantial computational overhead. In this work, we propose a refined security principle: only agents capable of deliberative System 2 reasoning may access untrusted documents. To evaluate this principle, we introduce novel metrics that quantify the discrepancy between misinformation detection and downstream influence. We then empirically compare state-of-the-art reasoning language models with standard language models across these metrics. Our results show that reasoning-capable models are substantially more robust to corrupted evidence, without requiring the strict isolation imposed by the Cordon Principle. These findings provide empirical support for our refined principle and suggest a more practical foundation for secure RAG system design.
