Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the critical challenge of preemptive unlearning in open-weight LLMs, aiming to prevent the acquisition of forbidden capabilities during downstream fine-tuning rather than removing them post-hoc. The authors demonstrate that existing retrospective unlearning methods are insufficient as forbidden-domain data can still induce gradients through internal pathways, enabling future acquisition. They propose a novel "gradient sealing" principle that effectively blocks these pathways by manipulating pre-activations, showing significantly improved resistance to downstream acquisition across multiple LLM families.
Forbidden capabilities can still be learned even when suppressed in current outputs, as internal gradients enable future acquisition, but gradient sealing effectively blocks these pathways.
Open-weight LLMs are released not only as fixed products but also as substrates for downstream fine-tuning. This openness, however, creates legal and ethical risks because users may misuse fine-tuning to instill illicit knowledge or enable hostile operations. Model providers therefore need apre-release defense against such acquisition, motivating the problem of preemptive unlearning. Unlike retrospective unlearning, which removes capabilities already present in a fixed model, preemptive unlearning seeks to prevent their acquisition under unseen attack data and future fine-tuning procedures. Despite its practical importance, this setting remains largely unexplored, presents distinct challenges, and is therefore the central focus of our work. We first verify that existing retrospective methods provide insufficient pre-release protection. Even when forbidden capabilities are suppressed in current outputs, forbidden-domain data can still induce gradients through internal pathways, enabling later acquisition. Motivated by this finding, we propose a gradient-sealing principle that blocks these pathways by pushing relevant pre-activations into the negative region, where ReLU-family activations exhibit zero or near-zero derivatives. Experiments across multiple LLM families demonstrate our stronger resistance to downstream acquisition than retrospective baselines, validating gradient sealing as an effective mechanism for pre-release protection.