Search papers, labs, and topics across Lattice.
This paper investigates the limitations of Gumbel-based inference verification in the context of adversarial attacks, revealing that the method significantly underperforms when faced with adversaries who control prompt distributions. The authors demonstrate that engineered prompts can exploit the model's output entropy, leading to a substantial increase in information leakage, with their strongest attack nearly doubling the bits leaked per token compared to benign prompts. This finding highlights the inadequacy of static thresholds for defense mechanisms, suggesting a need for dynamic calibration based on local token entropy.
Adversarial prompts can exploit Gumbel-based verification, nearly doubling information leakage and undermining static defense thresholds.
Gumbel-based inference verification bounds LLM weight exfiltration by only forgiving token choices that plausibly arise from honest GPU nondeterminism, reporting a>200x slowdown for a steganographic adversary under benign prompt traffic. This bound assumes a passive attacker; we show it degrades sharply against an adversary who instead controls the prompt distribution. Because the verifier's admissible-token-set size is driven by the model's own output entropy, prompts engineered to break grammatical and sub-word structure -- rather than benign conversational traffic -- widen that set and open a materially larger covert channel. Across six instruction-tuned models spanning 1B to 32B parameters and three random seeds, our strongest attack (character- and script-level disruption) roughly doubles bits leaked per token relative to benign prompts, cutting the slowdown factor to 60x - 118x. These results indicate that static, benign-traffic-calibrated thresholds are insufficient for this defense, and that jitter-forgiveness thresholds should instead be calibrated dynamically against local token entropy.
