Search papers, labs, and topics across Lattice.
This paper introduces Entropy-guided Token Weighting (ETW), a novel token-level unlearning regularizer designed to enhance the unlearning process in large language models (LLMs) by prioritizing informative tokens based on their predictive entropy. By leveraging the relationship between token informativeness and entropy, ETW effectively differentiates between function words and more informative content words, allowing for selective unlearning that maintains model utility. The results show that ETW outperforms existing methods by achieving effective unlearning while minimizing degradation in model performance.
Informative tokens can be selectively unlearned without sacrificing model performance, thanks to a novel entropy-based weighting mechanism.
Unlearning in large language models (LLMs) has emerged as a promising safeguard against adversarial behaviors. When the forgetting loss is applied uniformly without considering token-level semantic importance, model utility can be unnecessarily degraded. Recent studies have explored token-wise loss regularizers that prioritize informative tokens, but largely rely on ground-truth confidence or external linguistic parsers, which limits their ability to capture contextual information or the model's overall predictive state. Intuitively, function words like "the" primarily serve syntactic roles and are highly predictable with little ambiguity, but informative words admit multiple plausible alternatives with greater uncertainty. Based on this intuition, we propose Entropy-guided Token Weighting (ETW), a token-level unlearning regularizer that uses entropy of the predictive distribution as a proxy for token informativeness. We demonstrate that informative tokens tend to have higher entropy, whereas structural tokens tend to have lower entropy. This behavior enables ETW to achieve more effective unlearning while better preserving model utility than existing token-level approaches.
