Search papers, labs, and topics across Lattice.
This paper introduces PVDetector, a novel framework for detecting prompt injection (PI) attacks on purpose-specific large language model (LLM) agents by analyzing their hidden activation space for latent policy-violation (PV) concepts. By leveraging the intrinsic awareness of LLMs to recognize conflicts between user queries and predefined restrictions, PVDetector measures hidden-state alignment with these PV concepts, providing a more effective detection mechanism than existing input-output pattern analysis methods. Experimental results demonstrate that PVDetector achieves less than 1% false negative rate while maintaining minimal auxiliary overhead, significantly outperforming current state-of-the-art detection techniques.
LLMs can inherently recognize policy violations, and PVDetector exploits this capability to achieve unprecedented detection accuracy against prompt injection attacks.
Large language models (LLMs) are increasingly deployed as purpose-specific agents to handle domain-specific tasks such as customer service and code generation. These agents are expected to comply with not only generic safety guardrails but also purpose-specific restrictions tailored to their designated roles. Such additional restrictions enlarge the attack surface, particularly to prompt injection (PI) attacks. To defend against such attacks, existing detection methods primarily rely on analyzing input-output patterns, yet yield limited effectiveness. To address this limitation, we turn to analyzing the hidden activation space and discover that LLMs inherently retain latent policy-violation (PV) concepts when prompted with requests beyond their designated purpose. Particularly, PV concepts capture the semantics of conflicts between user queries and predefined restrictions, implicitly reflecting LLMs' intrinsic awareness of recognizing policy violations. Building on this insight, we propose PVDetector, a training-free framework that detects PI attacks during LLM inference by measuring hidden-state alignment with PV concepts, which are derived offline from the contrastive pairs of policy-violating and policy-compliant prompts. Experiments across multiple LLMs and datasets show that PVDetector achieves <1\% false negative rate with minimal auxiliary overhead, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/Claresigle/PVDetector .
