Search papers, labs, and topics across Lattice.
The paper introduces Attention Smoothing Unlearning (ASU), a novel framework for unlearning in large language models that addresses the limitations of existing methods by directly targeting lexical and semantic associations within the attention mechanism. ASU leverages self-distillation from a forget-teacher model, derived by increasing the softmax temperature of the original model's attention distributions, to flatten these distributions and suppress the reconstruction of memorized knowledge. Empirical results across various benchmarks (TOFU, MUSE, WMDP) and unlearning scenarios demonstrate that ASU achieves superior unlearning performance with minimal utility loss compared to baseline methods.
Forget sensitive data without sacrificing model coherence: Attention Smoothing Unlearning (ASU) leverages self-distillation with temperature-scaled attention to erase unwanted knowledge while preserving general language abilities.
Large Language Models are prone to memorizing sensitive, copyrighted, or hazardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoherent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), a principled framework that casts unlearning as self-distillation from a forget-teacher derived from the model's own attention. By increasing the softmax temperature, ASU flattens attention distributions and directly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization objective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most unlearning scenarios, delivering robust unlearning with minimal loss of model utility.
