Search papers, labs, and topics across Lattice.
This paper introduces ALIBI, an automated adaptive black-box attack framework that exploits adversarial comments to manipulate LLM-based vulnerability detectors without altering program behavior. By transforming real-world vulnerability-fixing commits into coding tasks, the study reveals that all evaluated detectors are highly susceptible, with attack success rates exceeding 90% and reaching 100% for one system. The findings highlight the critical need for security-aware designs in LLM-based tools, as current defenses offer limited protection against these adaptive attacks.
Adversarial comments can bypass LLM-based vulnerability detectors with over 90% success, exposing a critical vulnerability in AI-driven security tools.
Large language models are increasingly deployed for security-sensitive tasks such as vulnerability detection and code review. Their reliance on natural-language context embedded in source code exposes a previously underexplored attack surface: adversarial comments that can influence a detector's reasoning without changing program behavior. We study LLM-based vulnerability detectors against a new adversary: a coding agent that implements new functionality, deliberately introduces vulnerabilities, and strategically inserts adversarial source-code comments to evade detection. We present ALIBI, an automated adaptive black-box attack framework that generates and iteratively refines adversarial comments using detector reasoning and feedback. We transform real-world vulnerability-fixing commits into coding tasks and evaluate four representative LLM-based vulnerability detectors, ranging from specialized open-weight reasoning models to frontier multi-agent systems. All evaluated detectors are highly vulnerable: attack success rates exceed 90% across 125 real-world null-pointer dereference vulnerabilities, reaching 100% on one system. The framework also generalizes beyond this vulnerability class. Adversarial comments steering detector reasoning or fabricating external tool results prove most effective, while iterative refinement based on detector feedback further increases attack success. Finally, prompt-level defenses provide limited robustness against adaptive attacks, whereas architectural isolation and pre-detector comment sanitization substantially improve resilience. Our findings expose a fundamental attack surface in current LLM-based vulnerability detectors and motivate security-aware designs that carefully calibrate trust between natural-language context and program evidence.
