Search papers, labs, and topics across Lattice.
This paper introduces the AdaPop (Adaptive Popularity) method for unlearning in large language models (LLMs), which addresses the challenge of deeply ingrained popular facts that resist removal. By leveraging a popularity-dependent exponent and a dual-ascent controller, AdaPop dynamically adjusts the balance between forgetting and retaining information based on the frequency of training data. The results show that AdaPop significantly reduces the leakage of forgotten content compared to existing methods, achieving approximately 5x less leakage under paraphrased queries and 1.6x less under adversarial reformulations across multiple model families and benchmarks.
Popular facts are harder to forget, but AdaPop can effectively unlearn them while retaining essential knowledge, outperforming existing methods by a significant margin.
Popular facts are memorised more deeply during pretraining and resist removal longer than rare ones, yet existing LLM unlearning methods apply uniform gradient pressure regardless of training-data frequency. We propose the AdaPop (Adaptive Popularity) method, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy (e.g., Wikidata sitelinks, LLM-as-Judge), and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch. Across three model families and two benchmarks, AdaPop leaks ~5x less forgotten content than competing methods under paraphrased queries and ~1.6x less under adversarial reformulations. We support our analysis with internal metrics: under our method, forget-set hidden states move further from the pre-unlearning model's states than under other methods, while retain-set representations remain close.
