Search papers, labs, and topics across Lattice.
This paper introduces ConceptGuard, a benchmark designed to evaluate context-sensitive unlearning in Large Language Models (LLMs) by focusing on dual-use concepts that can be harmful or benign depending on context. The authors argue that traditional unlearning methods fail to adequately assess the ability to remove harmful knowledge while retaining beneficial information, highlighting the need for a more nuanced approach. Their findings reveal that existing unlearning techniques struggle with contextual separation and exhibit poor performance in maintaining concept-level control, underscoring a significant gap in current methodologies for ensuring safety in LLMs.
Unlearning in LLMs is not just about forgetting facts; it鈥檚 about mastering the nuanced balance of harmful and benign concept usage, and current methods fall short.
Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning, yet existing methods and benchmarks fail to evaluate this capability completely. Current approaches rely on disjoint forget and retain sets composed of independent facts, and measure success using simple and direct factual recall. This framing fails to capture a key requirement of unlearning, namely the ability to eliminate harmful behaviors while preserving benign and beneficial knowledge. We argue that effective unlearning must operate at the level of concepts, ensuring complete removal of unsafe applications while maintaining their correct and useful usage, thereby achieving conceptually meaningful and complete unlearning. To better evaluate unlearning techniques from such a practical viewpoint, we introduce the notion of dual-use concepts: concepts that can be used in both harmful and benign contexts. Building on these concepts, we construct a benchmark called ConceptGuard where forget and retain sets are explicitly complementary in concept usage. Our benchmark uniquely enables unlearning to be explored and gauged at the level of concepts, instead of sparse facts, and evaluation is intent-sensitive with the goal of maximizing contextual separation to promote safer behavior. We demonstrate that current unlearning techniques perform poorly under this setting, showing weak contextual separation alongside poor performance in ROUGE and concept-level metrics. Our results reveal strong forgetting-utility trade-offs, limited gains in contextual sensitivity, and poor consistency in concept-level control across methods, and provide ideas for unlearning approaches that better align with real-world safety requirements. Our dataset is publicly available.
