Search papers, labs, and topics across Lattice.
This study investigates the mechanisms of exact deletion from language-model memory, distinguishing between addressable influence removal and the challenges posed by shared recurrent state. By modifying the Gemma 3 model with support-vector memory and employing low-rank recovery techniques, the authors demonstrate that both decrement and retained-key refit methods yield nearly identical next-token outputs, achieving a median KL divergence of $5.4\times10^{-15}$ over multiple deletions. The findings reveal that exact deletion is contingent on the memory representation, allowing for effective subtraction of addressable records while enabling the replay of entangled writes.
Exact deletion from language-model memory can be achieved through clever memory representation, allowing models to effectively manage and amend their learned records.
Exact deletion from persistent language-model memory depends on how that memory represents a record. Addressable influence can be removed by algebraic decrement; influence transformed by later writes inside shared recurrent state requires rebuilding from before the write. We test this distinction in two pretrained models against explicit record-omitted references. First, we replace Gemma 3's global-attention layers with support-vector memory. After low-rank recovery at 1B, decrement and retained-key refit agree at the next-token output to median KL $5.4\times10^{-15}$ over 31 support-token deletions, with $+2.0\%$ perplexity relative to a matched fine-tune. A masked-refit proxy is indistinguishable from the never-ingested floor under elicitation, relearning, sampling, and LiRA attacks. At 4B and 12B, certificate ordering persists but utility cost rises to $11.2\%$ and $44.3\%$. Second, in a 48B Kimi Linear hybrid, additive writes admit a fixed decrement and diagonal decay a corrected one, whereas the delta rule makes $12$--$49\%$ of a record's contribution suffix-dependent. Checkpointed rewind-and-replay deletes real clinical records at contexts up to 18,842 tokens, matching never-ingested logits and all recurrent states bit for bit within a deterministic MLX implementation; replaying a correction provides exact amendment. Exact deletion is therefore a property of memory representation: subtract addressable records and replay entangled writes.
