Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates "revision traces," a privacy failure mode where LLMs executing redactions or edits in deliverables inadvertently expose withdrawn secrets by helpfully restating what was removed (e.g., commenting that a specific password was deleted). Analyzing 26,753 in-the-wild revision requests reveals traces in 8.8% of cases, prompting the creation of RevLeakBench to systematically evaluate six LLMs across conversational and agentic settings. Across evaluated models, ~50% of deliverables include revision traces and ~13% fully leak the withdrawn sensitive item, with leaks persisting in 36.4% of outputs even when models are explicitly informed that third parties will see the response.
LLMs asked to redact private data routinely sabotage their own edits, leaking the withdrawn secret in 13% of deliverables simply by helpfully stating what they just removed.
Large language model (LLM) assistants increasingly help users draft content for third-party recipients. During private drafting, the user or the model may introduce an item and later remove or replace it. The model may remove the item from the intended content but reveal it again when stating the edit. We call such statements revision traces. For example, after a user removes the password before sharing a configuration file, the model may delete it but leave a comment saying, "Removed the password 'No****4!' as requested." A third-party recipient who sees only the delivered file can therefore recover the withdrawn password from the comment. In an in-the-wild analysis of three public conversation corpora, we identify 26,753 revision requests, of which 2,363 (8.8%) leave revision traces. We study them in greater depth under controlled conditions by introducing RevLeakBench, a benchmark of 100 tasks across five scenarios with a conversation track and an agent track. We measure trace occurrence, withdrawn-item recovery, trace position, and required-content retention. Across six models, about half of the deliverables in both tracks state the edit after a revocation, and a reader that sees only the deliverable can recover the withdrawn item from about 13% of them. Telling the model that its entire reply will be forwarded to the recipient still leaves revision traces in 36.4% of the deliverables. We compare prompt defenses and a delivery boundary, and propose an output-side filter that sharply reduces recovery with little loss of required content. We believe our work can benefit efforts to understand and mitigate unintended disclosure in LLM interactions.