Search papers, labs, and topics across Lattice.
This paper introduces the Forget-Retain Alignment Gap (FRAG), a novel predictor for assessing the robustness of large language models (LLMs) against unlearning failures by evaluating the alignment of forget-critical and retain-critical weights. The authors demonstrate that traditional global weight-space distance metrics can be misleading, as they do not account for the structure of updates that impact relearning robustness. Their findings reveal that employing Forget-Retain Pruning (FRP) based on this alignment gap significantly enhances the ability of LLMs to maintain unlearned information, outperforming existing methods that rely solely on distance metrics.
Forget-Retain Alignment Gap reveals that the structure of weight updates, not just their distance, is key to preventing LLMs from relearning forgotten information.
Machine unlearning aims to make a model forget specific data, yet unlearned LLMs often fail to stay unlearned: brief fine-tuning can revive removed knowledge. Existing robustness predictors rely on global weight-space displacement, but distance alone can be misleading when random or destructive updates collapse performance. We argue that relearning robustness depends on update structure: robust unlearning should affect forget-critical weights while sparing retain-critical ones. We introduce the Forget-Retain Alignment Gap (FRAG), a training-free predictor that scores an update's forget-retain alignment without running a relearning attack, and separates selective from dense updates more reliably than global distance. Building on the forget-critical, retain-sparing principle, Forget-Retain Pruning (FRP) improves relearning robustness. Our results suggest that weight selectivity better explains robustness than distance alone. Code is available at https://github.com/Yi1-Chen/FRAG.
