Search papers, labs, and topics across Lattice.
This paper introduces an unsupervised method for detecting adversarial context documents in Retrieval Augmented Generation (RAG) systems, addressing a critical security vulnerability. The approach leverages generator activations, output embeddings, and entropy-based uncertainty to identify compromised documents without requiring labeled adversarial examples. Experiments demonstrate the effectiveness of these indicators and show that context summarization can improve detection performance, even without knowledge of the target prompt.
Unsupervised detection of adversarial attacks in RAG systems is possible using generator activations and uncertainty measures, even without knowing the target prompt.
Retrieval augmented generation systems have become an integral part of everyday life. Whether in internet search engines, email systems, or service chatbots, these systems are based on context retrieval and answer generation with large language models. With their spread, also the security vulnerabilities increase. Attackers become increasingly focused on these systems and various hacking approaches are developed. Manipulating the context documents is a way to persist attacks and make them affect all users. Therefore, detecting compromised, adversarial context documents early is crucial for security. While supervised approaches require a large amount of labeled adversarial contexts, we propose an unsupervised approach, being able to detect also zero day attacks. We conduct a preliminary study to show appropriate indicators for adversarial contexts. For that purpose generator activations, output embeddings, and an entropy-based uncertainty measure turn out as suitable, complementary quantities. With an elementary statistical outlier detection, we propose and compare their detection abilities. Furthermore, we show that the target prompt, which the attacker wants to manipulate, is not required for a successful detection. Moreover, our results indicate that a simple context summary generation might even be superior in finding manipulated contexts.
