Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
To isolate vulnerabilities uniquely introduced by visual token compression rather than inherited from base models, this paper formalizes compression-specific failures (CSFs) using a paired counterfactual attribution framework and introduces CIRA, an attack exploiting this dynamic. Operating strictly on the vision encoder without access to the language model, compression algorithm, prompts, or target budget, CIRA optimizes image perturbations to manipulate token retention priorities while preserving displaced representations. Across 12 dataset-compressor configurations, CIRA achieves a 20.35% mean compression-specific failure rate while maintaining full-token attack success at only 6.92%, demonstrating that compression mechanisms introduce severe, distinct attack surfaces.
Visual token pruning creates a silent vulnerability vector: lightweight perturbations can manipulate token selection to blind compressed vision-language models while leaving uncompressed inference completely intact.
Visual token compression reduces the inference cost of Large Vision-Language Models (LVLMs). However, aggregate robustness measures do not reveal whether a particular adversarial failure is induced by compression or inherited from the underlying model. We define a compression-specific failure (CSF) as an adversarial input that remains correct under full-token inference but fails after compression, casting compression-induced risk as a paired failure attribution problem. Within a controlled diagnostic cohort, counterfactuals show that retained-set allocation causally changes compressed correctness and reveal a negative association between recovery and representation drift in displaced evidence. Motivated by these findings, we propose CIRA, a Compression-Induced Risk Attack for Large Vision-Language Models. Under a vision-encoder white-box setting, CIRA optimizes image perturbations through encoder-side objectives that manipulate token priorities across candidate compression budgets while preserving displaced evidence. CIRA uses no downstream questions or labels and requires no access to the language model, deployed compressor, or exact compression budget. Across 12 dataset-compressor settings evaluated at four budgets, CIRA achieves a mean CSFR of 20.35% while limiting full-token attack success to 6.92%, with similar behavior on additional LVLM families. A cross-view selection-stabilization defense substantially suppresses CIRA, although Adaptive CIRA partially restores its effectiveness. These results show that compression-specific failures persist under restricted access and support paired evaluation of full-token and compressed inference for attributing risk to visual-token compression.