Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces SIEVE, a novel framework for selective unlearning in vision-language models (VLMs) that addresses the challenge of removing personally identifiable information (PII) while retaining other knowledge about the same individual. SIEVE directly targets attention-value representations by suppressing forget-example attention to zero and preserving retain-example representations via reference matching, augmented with sequence-level supervision. Experimental results demonstrate SIEVE's state-of-the-art performance in targeted unlearning across various VLM configurations, achieving significant forgetting with minimal degradation of retained utility.
Unlearning PII from VLMs is now feasible without sacrificing retained knowledge by directly manipulating attention-value representations.
The ability of vision-language models (VLMs) to associate visual identities with biographical information creates a need for selective unlearning of personally identifiable information (PII) while preserving permitted knowledge about the same individual. This setting is challenging because both sensitive and retained information can share the same visual inputs and intermediate representations. We introduce SIEVE, a simple and effective framework for selective VLM unlearning. SIEVE directly regularizes attention-value representations while also controlling model outputs. SIEVE suppresses attention values for forget examples toward a constant zero, while preserving retain-example representations by matching them to a frozen reference model. These objectives are combined with sequence-level forget and retain supervision, enabling targeted forgetting without largely affecting retained knowledge. Extensive experiments show that SIEVE achieves state-of-the-art performance on unlearning with multiple model-modality settings, while maintaining competitive retained utility. Ablation studies further show that value suppression and negative cross-entropy contribute complementary forgetting signals, while reference-based value matching substantially reduces utility degradation. These results demonstrate that attention values provide an effective intervention point for selective multimodal unlearning when sensitive and retained knowledge are closely related.