Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of removing specific individual information from multimodal large language models (MLLMs) without relying on a retain set, which is difficult to obtain post-deployment. By framing the problem as a localization issue, the authors identify early-to-mid decoder MLPs as the optimal layers for unlearning, allowing for targeted modifications that minimize disruption to shared visual-language computations. The proposed Pathway-Aware Visual-attribute Anchoring (PAVA) method achieves the best forget-retain trade-off among forget-set-only techniques while maintaining competitive performance against retain-based approaches on benchmark datasets.
Forgetting individual information in MLLMs can be achieved without compromising model performance, thanks to a novel method that localizes updates to specific layers.
Removing a specific individual's information from multimodal large language models (MLLMs) is often needed after deployment, but existing methods rely on a retain set, which is hardest to obtain at that point, and rebuilding it recreates the privacy exposure that unlearning aims to remove. Forgetting from the forget set alone instead damages the shared visual-language computation, harming perception. We cast retain-free unlearning as a localization problem: causal tracing, weight transplant, and Fisher overlap all point to early-to-mid decoder MLPs as the layers where identity information is stored and, unlike other module families, can be modified without substantially disrupting vision. We turn this into Pathway-Aware Visual-attribute Anchoring (PAVA), which confines updates to these layers and pairs a forget loss with a visual-attribute anchor that preserves image-grounded behavior by distilling the model's own pre-unlearning answers from the forget images alone. On MLLMU-Bench and ReMem, PAVA gives the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.