Search papers, labs, and topics across Lattice.
This paper investigates the security vulnerabilities in GUI visual grounding models that generate screen coordinates as sequences of digit tokens, revealing that small changes in digit predictions can lead to significant misplacements of executed clicks. The authors introduce MissClick, a white-box adversarial attack with two tailored objectives: MissClick-U for maximizing coordinate displacement in untargeted attacks and MissClick-T for optimizing targeted attacks based on place-value structures. Experimental results demonstrate that MissClick significantly outperforms existing attacks, achieving untargeted success rates of up to 75.07% and targeted success rates of up to 62.67% across various platforms.
Small digit changes can lead to large click misplacements, and MissClick exploits this to achieve unprecedented attack success rates against GUI grounding models.
Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks. The security implications of this coordinate generation process have been largely overlooked. We observe that each coordinate digit is predicted as a categorical token, yet after parsing, changing a hundreds-place digit by one changes the corresponding numerical coordinate component by 100 units, which can induce a large displacement of the executed click. This observation motivates attack objectives that account for the numerical and place-value structure of coordinate outputs rather than treating them as ordinary text. Moreover, untargeted and targeted attacks impose different success conditions--displacing the click outside the correct region versus into an attacker-specified region--and therefore benefit from different objectives. We propose MissClick, a simple and effective white-box adversarial attack with two goal-specific objectives: MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted hijacking. Compared with existing attacks against GUI grounding models on OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07\% and 72.93\% (+16.62 and +30.72 pp), and MissClick-T achieves targeted success rates of 44.86\% and 62.67\% (+31.73 and +47.06 pp). Attack objective comparison further shows that soft-coordinate displacement yields the highest untargeted attack success rate, whereas place-weighted target-digit optimization yields the highest targeted attack success rate, revealing distinct objective preferences for the two attack goals.
