Search papers, labs, and topics across Lattice.
This paper introduces MemCatalyst, a novel set of data poisoning tools designed to enhance data auditing capabilities for Vision-Language Models (VLMs) by leveraging two strategies: Poisoning Text (PT) and Poisoning Image (PI). By inducing VLMs to over-learn inconsistencies between image features and textual semantics, MemCatalyst significantly improves membership inference performance while requiring only a small number of poisoned samples. The results show a marked increase in AUC scores for membership inference across different VLM architectures, demonstrating the method's effectiveness in a black-box setting without compromising model performance.
MemCatalyst reveals that targeted data poisoning can drastically boost membership inference accuracy in Vision-Language Models with minimal resource expenditure.
Vision-Language models (VLMs) achieve outstanding performance largely due to the amount of training data available on the internet. At the same time, data holders (e.g., artists) urgently need to determine whether their data has been used for model training without authorization, which concerns both intellectual property rights and personal privacy. Data auditing, particularly through membership inference (MI), has attracted attention as a direct tool. This work proposes MemCatalyst, a set of data poisoning tools, aiming to amplify the data auditing performance on VLMs. MemCatalyst employs two strategies: Poisoning Text (PT) and Poisoning Image (PI). MemCatalyst forces VLMs to over-learn specific inconsistencies between image features and textual semantics during training, thereby increasing their susceptibility to membership information auditing. Crucially, the transferability of poisoned samples across different VLM architectures is demonstrated to be effective in the black-box setting. Extensive evaluations using five state-of-the-art data audits on two prominent VLMs demonstrate that MemCatalyst markedly enhances MI AUC scores with a minimal budget of poisoned samples, while maintaining a negligible impact on model performance.
