Search papers, labs, and topics across Lattice.
This paper introduces a causal auditing framework to evaluate the effectiveness of deletion-based unlearning in Limited Memory Language Models (LMLMs) by analyzing how deleted facts persist through various retrieval mechanisms. The authors conducted extensive experiments on 12,228 alias-closure deletions across multiple databases, revealing that parametric leakage is negligible while the majority of post-deletion correctness stems from retrieval-mediated sources. Their findings indicate that the unlearning capabilities of LMLMs are largely determined by the database state rather than the model's inherent memory, challenging assumptions about model retention of deleted information.
Unlearning in LMLMs is more about database management than model architecture, with up to 13.6% of deleted facts surviving through retrieval artifacts.
Limited Memory Language Models (LMLMs) externalize factual knowledge to a database to enable deletion-based unlearning without retraining. Existing evaluations measure post-deletion correctness in aggregate and cannot tell whether a deleted fact persists through residual parametric memory, alternative retrieval paths, or near-neighbor retrieval artifacts. We propose a causal auditing framework that holds the model fixed and varies the database state at inference time across three interventions: FULL, DEL-ON, and DEL-OFF. The framework decomposes post-deletion behavior into parametric leakage L(f), retrieval-mediated correctness R(f), and a retrieval artifact rate grounded in the inference-time retrieval trace. We apply it to 12,228 alias-closure deletions across thirteen databases, including four adversarial topologies (Base, Alias, Noise, Collision) we construct in three domains, and six prompt formulations. Parametric leakage is near zero in every variant and every prompt style: the model rarely returns the deleted answer in the absence of retrieval. The residual that does survive lives in the retrieval graph: retrieval-mediated correctness and the retrieval artifact rate match within rounding everywhere, so post-deletion correctness is, in our audit, predominantly reconstituted from near-neighbor retrieval. This residual ranges from 0.7% on the released LMLM database to 13.6% on the most adversarial variant, and prompt formulation does not independently control how much of a deleted fact survives. These results suggest that, for this class of LMLM and deletion procedure, the unlearning boundary is drawn primarily by the database administrator rather than by the model.
