Search papers, labs, and topics across Lattice.
This paper introduces a black-box Membership Inference Attack (MIA) against VideoLLMs, exploiting the hypothesis that member videos induce sharper, more brittle generation behavior across different decoding temperatures. The attack queries the VideoLLM at varying temperatures and measures the semantic drift in the generated text, incorporating video-aware difficulty features to enhance the inference. Experiments on \texttt{LLaVA-Video-7B-Qwen2-Video-Only} demonstrate the vulnerability of VideoLLMs to this MIA, achieving an AUC of 0.68 and accuracy of 0.63.
VideoLLMs leak training data: a novel black-box attack recovers membership with surprisingly high accuracy (AUC=0.68) by probing generation brittleness across temperatures.
Video large language models (VideoLLMs) are increasingly trained or instruction-tuned on large-scale video--text corpora collected from heterogeneous sources, raising an immediate privacy question: can an external auditor determine whether a particular video was used during training? While membership inference attacks (MIAs) have been studied extensively for classifiers and, more recently, for text and image generation models, the VideoLLM setting remains unexplored. This setting is challenging because black-box auditors observe only generated text, whereas the membership signal is entangled with video-specific factors such as motion complexity and temporal span. In this paper, we present a black-box MIA targeting VideoLLMs that couples temperature-perturbed generation with video-aware difficulty features. Our key intuition is that member samples tend to induce sharper, more brittle generation behavior across decoding temperatures, and that this signal should be interpreted jointly with the intrinsic difficulty of the queried video. Concretely, we query the target model at low and high temperatures, measure the semantic drift between the resulting texts. We evaluate the attack against \texttt{LLaVA-Video-7B-Qwen2-Video-Only} and achieve a member inference AUC of 0.68 and accuracy of 0.63. These results demonstrate that Video-LLMs are vulnerable to black-box membership inference attacks, highlighting an urgent need for the community to systematically evaluate and mitigate privacy risks in VideoLLMs.
NUS