Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of inadvertent context leakage in language models, revealing that sensitive user data can be reconstructed from benign outputs even when direct extraction is refused. Through controlled experiments on eight proprietary models, the authors demonstrate that 2-digit secrets can be reconstructed with near-perfect accuracy and 4-digit secrets with 82% accuracy, highlighting a concerning correlation between model capability and leakage. The findings indicate that stronger instruction-following in models inadvertently increases their sensitivity to in-context secrets, posing significant risks for user privacy.
Language models can reconstruct sensitive user data from innocuous outputs, with 2-digit secrets retrieved almost perfectly and 4-digit secrets at 82% accuracy, raising serious privacy concerns.
For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82\% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.
DeepMind
Meta AI