Search papers, labs, and topics across Lattice.
This study systematically investigates prompt leaking attacks in over 1,200 real-world LLM-based applications, revealing that more than 80% of deployments are susceptible to leaking sensitive system prompts under adversarial queries. The research identifies the mechanisms behind these leaks, particularly attention drift caused by query-key alignment bias and softmax amplification, which undermine existing defenses without enhancing usability. To address this, the authors introduce AREA, a novel defense that re-anchors model attention using an optimizable soft prompt, achieving both enhanced leakage resistance and a 33% improvement in usability compared to state-of-the-art methods.
Over 80% of real-world LLM applications leak sensitive prompts, but a new defense, AREA, not only mitigates this risk but also boosts usability by over 33%.
Large language model (LLM)-based applications rely on system prompts to encode core logic and developer-defined constraints, making these prompts important intellectual property. However, system prompts are vulnerable to prompt leaking attacks. Although prior work has shown such attacks in controlled settings, their prevalence, causes, and defenses in real-world deployments remain unclear. This paper presents a systematic study of prompt leaking in real-world LLM-based applications. We measure 1,200 applications across six major commercial platforms and find that over 80% of deployments leak system prompts under realistic adversarial queries, sometimes exposing sensitive information such as third-party API keys. We also show that existing defenses often fail to prevent leakage without degrading usability. To explain these failures, we conduct an attention-level mechanistic analysis and identify attention drift, where query-key alignment bias and softmax amplification cause LLMs to progressively ignore defensive constraints. Guided by this insight, we propose AREA, a practical defense that re-anchors the model's attention using an optimizable soft prompt. Experiments and real-world case studies show that AREA matches the leakage resistance of state-of-the-art defenses while improving average usability by over 33% and reducing optimization overhead by nearly 3x. Our responsible disclosure led two affected vendors to classify these leaks as medium-severity vulnerabilities.
