Search papers, labs, and topics across Lattice.
This paper introduces BLADE, a bilevel framework designed to enhance the robustness of LLM unlearning by addressing common pitfalls such as unbounded forget losses and fixed-weight balancing. By employing a clamped-entropy forget loss, an asymmetric augmented Lagrangian, and a bilevel structure focused on LoRA adapters, BLADE achieves significant improvements in model coherence and stability during unlearning processes. The method outperforms leading baselines across multiple benchmarks, achieving average composite score increases of 6% on TOFU, 9% on MUSE Books, and 7% on KnowUndo, while maintaining stability under scaling and repeated unlearning steps.
BLADE achieves unprecedented stability in LLM unlearning, outperforming leading methods by up to 9% while remaining robust under extreme scaling and repeated applications.
Existing LLM unlearning methods struggle with robustness: unbounded forget losses degrade model coherence, fixed-weight balancing cannot adapt as retain difficulty shifts mid-training, and methods that work on one benchmark falter under scaling or repeated application. We propose BLADE, a constrained bilevel framework whose three mechanisms give smooth, predictable control over the optimization landscape: a clamped-entropy forget loss whose gradient is exactly zero once a token reaches sufficient uncertainty; an asymmetric augmented Lagrangian that permanently ratchets retain protection after any violation; and a bilevel structure confined to LoRA adapters that repairs retain damage before each forgetting step. BLADE dominates across three benchmark families, improving average composite scores over the strongest baselines by $6$% on TOFU, $9$% on MUSE Books, and $7$% on KnowUndo, and it remains stable under $4\times$ scaling and $4$ sequential unlearning steps on MUSE News where the best competing method collapses entirely.
