Search papers, labs, and topics across Lattice.
This paper investigates the privacy vulnerabilities associated with collaborative inference in Large Vision-Language Models (LVLMs), revealing that intermediate hidden states can leak identifiable private information despite the intention to maintain privacy through raw input withholding. The authors introduce RASR, a novel multimodal reconstruction attack that effectively reconstructs both images and text from hidden states, achieving a significant reduction in reconstruction error and high token accuracy. The findings highlight a critical gap in privacy protections for LVLMs, demonstrating that sensitive visual and textual information can be recovered from deep-layer hidden states, thus raising alarms about the security of collaborative inference systems.
Privacy-sensitive visual and textual information can be recovered from deep-layer LVLM hidden states, exposing significant risks in collaborative inference.
Collaborative inference deploys Large Vision-Language Models (LVLMs) by partitioning computation between edge devices and the cloud. While withholding raw inputs supposedly ensures privacy, transmitting intermediate hidden states exposes a critical attack surface. However, it remains unclear whether deep-layer LVLM hidden states retain recoverable private information, given that visual content has been projected into the language embedding space. To address this concern, we theoretically analyze LVLM hidden-state recoverability and show that, under regularity assumptions and a positive semantic--nuisance margin, privacy-relevant visual semantics remain identifiable and stably recoverable. Motivated by this analysis, we propose RASR, a novel coarse-to-fine multimodal reconstruction attack. RASR obtains initial image and text reconstructions through modality-specific inverse paths that follow their respective forward processing pipelines in reverse, and then uses hidden-state consistency to refine both reconstructions. Evaluations on Qwen3-VL-8B-Instruct and LLaVA-1.5-7B across five datasets demonstrate that RASR reduces image reconstruction MSE by \(\sim\)50\% compared to the strongest baselines, while achieving up to 99\% token accuracy for text recovery. These results show that privacy-sensitive visual and textual information can be recovered even from deep-layer LVLM hidden states, exposing the privacy risks of collaborative inference.
