Search papers, labs, and topics across Lattice.
The paper introduces an automated framework to evaluate the vulnerability of LLM system instructions to encoding attacks, where extraction requests are disguised as structured output tasks. They found high success rates (>0.7) in extracting sensitive information across four common models and 46 system instructions, even when direct extraction requests are blocked. A mitigation strategy using one-shot instruction reshaping with Chain-of-Thought reasoning shows that subtle changes in wording can significantly reduce attack success.
LLMs readily leak sensitive system instructions when extraction requests are disguised as encoding tasks, even when direct requests are blocked.
System Instructions in Large Language Models (LLMs) are commonly used to enforce safety policies, define agent behavior, and protect sensitive operational context in agentic AI applications. These instructions may contain sensitive information such as API credentials, internal policies, and privileged workflow definitions, making system instruction leakage a critical security risk highlighted in the OWASP Top 10 for LLM Applications. Without incurring the overhead costs of reasoning models, many LLM applications rely on refusal-based instructions that block direct requests for system instructions, implicitly assuming that prohibited information can only be extracted through explicit queries. We introduce an automated evaluation framework that tests whether system instructions remain confidential when extraction requests are re-framed as encoding or structured output tasks. Across four common models and 46 verified system instructions, we observe high attack success rates (> 0.7) for structured serialization where models refuse direct extraction requests but disclose protected content in the requested serialization formats. We further demonstrate a mitigation strategy based on one-shot instruction reshaping using a Chain-of-Thought reasoning model, indicating that even subtle changes in wording and structure of system instructions can significantly reduce attack success rate without requiring model retraining.
