Search papers, labs, and topics across Lattice.
This study investigates the complexities of reward specification in GRPO-based LLM unlearning, focusing on the dual objectives of suppressing target-specific knowledge while maintaining non-target utility. By comparing four distinct reward designs within a controlled LoRA-GRPO RWKU framework, the authors reveal that optimization success does not necessarily align with effective behavioral unlearning, highlighting discrepancies between various evaluation metrics. The findings indicate that reward-hacking and benchmark limitations can obscure true model performance, emphasizing the need for more nuanced evaluation strategies in LLM unlearning tasks.
Optimization success in LLM unlearning can be misleading, as different evaluation metrics may yield conflicting insights about model behavior.
Practical LLM unlearning is usually evaluated through two objectives: suppress target-specific knowledge and preserve non-target utility. In generative QA, this leaves a third behavior underspecified: when a target-adjacent prompt admits a broader answer without target-specific leakage, the model should answer at that level rather than leak, evade, or refuse. We study this specification problem in a controlled LoRA-GRPO RWKU setting, comparing four reward designs that span lexical suppression, anti-refusal shaping, rubric-based broad answering, and an explicit refusal contrast, with and without SFT warm-up. The experiments show that optimization success is not equivalent to behavioral unlearning: RWKU forget scores, held-out completion audits, terminal training-rollout audits, and training dynamics can point to different conclusions. We trace these disagreements to reward-hacking endpoints, policy-support limits in GRPO, benchmark probes that miss endpoint changes, and rewards that can select broad-topic answering with low semantic leakage during optimization.
