Search papers, labs, and topics across Lattice.
This paper introduces a novel benchmark called "Leak-Resistant Unlearning" to evaluate the effectiveness of machine unlearning methods in large language models (LLMs) against multi-hop reasoning and recovery attacks. The authors demonstrate that existing unlearning methods are susceptible to knowledge leakage through complex reasoning paths and can be partially compromised by lightweight post-unlearning adaptations. Their findings reveal significant vulnerabilities in current approaches, highlighting the need for improved strategies to ensure robust knowledge removal in LLMs.
Existing unlearning methods can leak sensitive knowledge through multi-hop reasoning paths, exposing a critical vulnerability in LLMs.
Benchmarking machine unlearning methods is critical to understand whether sensitive knowledge is removed from large language models (LLMs) or not. Current unlearning benchmarks include mainly single-hop questions and a narrow set of multi-hop questions. Although effective, they still face two challenges. (1) Knowledge is not isolated, whereby diverse multi-hop reasoning paths can potentially induce knowledge leakage than normal queries. (2) Unlearning may be fragile: unlearned knowledge can be partially recovered through recovery attacks such as lightweight post-unlearning adaptation, making static evaluation insufficient. Therefore, in this paper, we introduce \unlearning as a novel benchmark to understand robust LLM knowledge removal across diverse reasoning paths and recovery attacks. We experiment with this benchmark on 3 models, 6 unlearning methods, and 2 carefully curated datasets. Results show that existing methods are vulnerable to multi-hop reasoning paths and recovery attacks. We further explore the trade-off among forget quality, robustness, and model utility for LLM unlearning.
Tsinghua AI