Search papers, labs, and topics across Lattice.
This study empirically tests the intuition that representational entanglement complicates unlearning in neural networks by employing Selective Gradient Masking (SGTM) to create a suite of language models with varying levels of disentanglement between biology and non-biology knowledge. The results reveal that more disentangled models achieve significantly better retain-forget trade-offs, with the most disentangled models showing up to four times lower retain costs at fixed forgetting levels. This provides direct evidence that entangled representations contribute to collateral damage during unlearning, validating long-held beliefs in interpretability research.
Disentangled representations can reduce unlearning collateral damage by up to 4x, challenging the assumption that entanglement is merely a theoretical concern.
A long-held intuition in interpretability research is that representational entanglement, the sharing of structure between knowledge domains in a neural network, makes unlearning harder. While the intuition is widespread, it has never been directly tested in a controlled experiment. We present a way to do so: by repurposing Selective Gradient Masking (SGTM), we train a suite of six 254M-parameter language models on English Wikipedia with graded levels of disentanglement between biology and non-biology knowledge. Applying three standard unlearning methods to every model in the suite, we find that more disentangled models consistently achieve better retain-forget trade-offs: at a fixed level of forgetting, the most disentangled models incur roughly $4\times$ lower retain cost under two of the three methods, and $1.3\times$ lower under the third. Because our intervention changes only the model, not the data or the unlearning algorithm, this is direct evidence that representational entanglement is one of the causes of collateral damage in unlearning, as interpretability researchers have long suspected. A similar design could be used to test other structural claims from interpretability.
