Search papers, labs, and topics across Lattice.
This study investigates the impact of reward design on the efficiency of reinforcement unlearning in language models, addressing the limitations of sparse binary rewards that hinder convergence. By introducing a reward decomposition framework and two novel reward functions鈥攁n exponential reward and a PageRank-inspired reward鈥攖he authors demonstrate significant improvements in unlearning speed and effectiveness. Experiments on the Real World Knowledge Unlearning benchmark reveal that these new reward functions can achieve similar forgetting performance up to three times faster than traditional binary rewards while maintaining model utility.
Reward design is a game-changer in reinforcement unlearning, enabling models to forget knowledge up to three times faster without sacrificing performance.
Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act. Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem, where models are optimized against verifiable rewards computed directly from their outputs. However, existing methods rely on sparse binary rewards that provide minimal learning signal, indicating only whether forbidden content was avoided, and limiting convergence speed. In this paper, we study how reward design affects unlearning efficiency within the Reinforcement Unlearning (RUL) framework. We introduce a principled reward decomposition framework that decouples verifiability from sparsity, and propose two new reward functions: an exponential reward that provides graded penalties based on the count of forbidden-concept occurrences, and a PageRank inspired reward that weights penalties by semantic importance. We conduct experiments on the Real World Knowledge Unlearning (RWKU) benchmark, demonstrating that both rewards consistently outperform the binary setting, while reaching similar forgetting performance up to $3\times$ faster and preserving general model utility. Our results show that reward design is a key driver of unlearning efficiency offering a practical path toward scalable and efficient machine unlearning.
