Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of overthinking in Large Vision-Language Models (LVLMs) used in robotic systems, where excessive reasoning leads to significant inference delays. By embedding carefully crafted scene text into the visual input, adversaries can trigger these overthinking behaviors, resulting in slowdowns that pose safety risks for robotic operations. The authors develop a three-stage framework to identify and validate transferable triggers, achieving slowdown ratios exceeding 6.96x across multiple LVLMs, highlighting the vulnerability of these systems to adversarial manipulation.
Adversaries can exploit overthinking in LVLMs to induce inference delays of up to 6.96x, jeopardizing robotic safety and performance.
Large Vision-Language Models (LVLMs) have been increasingly integrated into robotic systems. However, these models may exhibit overthinking behaviors, where they generate excessively long reasoning traces, incurring an excessive inference time. This overthinking behavior poses a serious risk to robotic systems, as the adversary can deliberately trigger overthinking to slow down the decision making of a victim robotic system, causing a variety of safety issues (i.e., an overthinking-induced slowdown attack). To initiate this attack, an adversary can embed carefully crafted, human-readable scene text into the visual scene observed by a victim robotic agent, causing significant inference delays even under a strict black-box setting. Therefore, the embedded scene text serves as a significant "trigger" for the attack. This work systematically identifies and validates transferable triggers of overthinking in robotic systems by introducing a three-stage framework. First, we construct a diverse corpus of reasoning-intensive scene text and extract overthinking-correlated lexical features from short response prefixes. Second, we perform an efficient black-box search guided by a prefix-based proxy score while selectively confirming a small set of top candidates with full latency measurements. Third, we evaluate black-box transfer using a fixed pool of triggers on unseen images and multiple LVLMs, reporting latency amplification and attack success rates under standard thresholds. Across three representative LVLMs, all triggers yield slowdown ratios greater than 1.0x, with the strongest single-trigger case reaching 6.96x. The physical printing of the text trigger still causes up to 4.74x latency amplification. These results demonstrate that our discovered triggers are transferred between multiple LVLM models and consistently cause significant slowdowns in robotic systems.
