Search papers, labs, and topics across Lattice.
This paper introduces FAIRGET, a benchmark for evaluating the effectiveness of unlearning in multimodal large language models (MLLMs) under imbalanced demographic conditions, addressing a gap in existing methodologies that typically assume uniform distribution of unlearning requests. The authors also present FAUN, an innovative unlearning algorithm that utilizes a bias-aware activation steering mechanism to ensure fairness while effectively removing personal data. Experimental results indicate that FAUN outperforms existing methods in both unlearning quality and fairness, highlighting the importance of considering demographic imbalances in model training and unlearning processes.
Unlearning can inadvertently reinforce biases if demographic requests are imbalanced, but FAUN ensures fairness while effectively removing sensitive data from MLLMs.
Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations. To evaluate unlearning effectiveness in multimodal large language models (MLLMs), prior works fine-tune models on fictitious identities, simulating unlearning requests on subsets of these IDs, which are typically uniformly distributed. However, in realistic scenarios, people from different demographic groups may request to be unlearned at different frequencies, potentially altering the model's internal beliefs for these groups and leading to biased behaviors. To fill this gap, we propose FAIRGET, the first Visual Question Answering benchmark that evaluates unlearning under unbalanced, realistic, forget requests. These requests are designed to simulate multiple realistic scenarios, ranging from simple to challenging settings, that lead to biased unlearned models if fairness is not accounted for. Additionally, we propose FAUN, the first unlearning algorithm for MLLMs that forgets unlearning data while preserving model fairness. FAUN exploits a bias-aware activation steering mechanism to unlearn identities while accounting for the unbalanced nature of the forget data. Experiments on FAIRGET and the established FIUBench demonstrate our method's superiority both in unlearning quality and fairness.
