Search papers, labs, and topics across Lattice.
This paper introduces Stochastic Meta-Unlearning (SMU), a bilevel framework designed to enhance unlearning in vision-language models (VLMs) by leveraging VLM-level feedback. The authors reveal that traditional text-only feedback is insufficient for effective unlearning, as certain targets can be recovered when visual information is included, highlighting the complexity of multimodal unlearning. Experimental results demonstrate that SMU significantly improves the forget-retain trade-off, achieving a 10.52-point reduction in average Forget accuracy while enhancing Retain and Test accuracy by 20.10 and 17.01 points, respectively, across multiple datasets and models.
VLM-level feedback can transform language backbone unlearning from unreliable to robust and transferable, achieving unprecedented performance gains.
Machine unlearning for vision-language models (VLMs) remains underexplored. Unlike language models, VLMs combine a language backbone with visual components, which makes unlearning more complex. There is a surprising phenomenon when moving from single-modality unlearning to VLM unlearning: a target forgotten by the standalone language backbone can still be recovered when image information is given to the full VLM. This shows that text-only feedback is not enough for reliable VLM unlearning. Motivated by this observation, we propose Stochastic Meta-Unlearning (SMU), a bilevel framework that uses VLM-level feedback to learn an unlearning-ready initialization. In the inner loop, SMU applies a few unlearning steps to the language backbone using text data. In the outer loop, SMU recomposes the updated backbone with the frozen VLM and evaluates forgetting and utility at the VLM level. This design makes the unlearning update aware of the final multimodal behavior, while still keeping the update local to the language backbone. Experiments on two VLMs, two multimodal meme datasets, and three baselines show that SMU achieves the best overall forget-retain trade-off. Compared with the strongest baseline for each metric, SMU reduces average Forget accuracy by 10.52 points and improves average Retain and Test accuracy by 20.10 and 17.01 points, respectively. More importantly, SMU also transfers to new forgetting targets and to different meta-test unlearning methods. These results suggest that VLM-level feedback can make language-backbone unlearning more reliable and more transferable for VLMs.
