Search papers, labs, and topics across Lattice.
This paper uncovers a significant vulnerability in Large Language Models (LLMs) that arises from the use of Grammar-Constrained Decoding (GCD), which is intended to enhance the reliability of code generation. The authors introduce a novel jailbreak attack, CodeSpear, that exploits GCD to compel LLMs to generate malicious code, demonstrating an increase in attack success rates by over 30 percentage points compared to existing methods. To counter this threat, they propose CodeShield, a safety alignment technique that effectively mitigates the risks posed by GCD while maintaining the model's utility in generating benign outputs.
GCD, designed to enhance code reliability, can paradoxically serve as a vulnerability that enables LLMs to produce malicious code through a novel attack called CodeSpear.
Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.
