Search papers, labs, and topics across Lattice.
This paper introduces ADU, a novel framework for unlearning in large language models that shifts the focus from simple token erasure to a more nuanced approach of contextual attention-pathway decoupling. By leveraging the functional differences between local and global attention heads, ADU effectively suppresses attention mass on sensitive information while maintaining linguistic integrity and overall model utility. The results demonstrate that ADU outperforms existing methods, achieving a Forget Quality of 0.93 on the TOFU benchmark and preserving an average of 92.9% model utility compared to 81.9% for baselines.
Unlearning sensitive information in LLMs can be achieved without sacrificing performance, with ADU preserving up to 98% of model utility while effectively forgetting.
Large language models (LLMs) require effective unlearning to address privacy regulations and safety concerns. However, achieving precise forgetting without compromising general utility remains challenging. Existing sequence- and token-level methods penalize target outputs without modeling their context-dependent retrieval paths, which can disrupt linguistic structure or suppress benign knowledge. We present ADU, a fine-grained, training-based framework that shifts unlearning from token erasure to contextual attention-pathway decoupling. Exploiting the functional distinction between local and global attention heads, ADU identifies preplan positions that retrieve persistent sensitive anchors and fixes their candidate paths under the original model. It then trains attention-projection adapters to suppress attention mass along these paths while preserving local-attention structure and retain-set language modeling. Post-training activation exchange tests whether the modified attention-output module transmits the learned forgetting effect. ADU achieves the strongest aggregate performance among evaluated baselines on the TOFU and WMDP benchmarks, including a Forget Quality of (0.93) on TOFU. It preserves 87--98% of model utility (92.9% on average versus 81.9% for baselines) while reducing side effects in benign contexts.
