Search papers, labs, and topics across Lattice.
This study evaluates the safety of AI-generated C++ code compared to human-written code using a comprehensive benchmark called VULBENCH-CPP, which encompasses 8,918 programs and employs four verification tiers: functional testing, static analysis, dynamic analysis, and bounded model checking. The findings reveal that AI-generated code is approximately twice as likely to incur confirmed runtime violations, despite appearing equally safe under static analysis, highlighting the inadequacy of relying solely on static methods for security assessments. This discrepancy underscores the necessity for multi-tier verification approaches to accurately assess the safety of AI-generated code in memory-unsafe languages like C++.
AI-generated C++ code is twice as likely to trigger runtime violations compared to human-written code, challenging the reliability of static analysis alone for safety evaluations.
Large language models increasingly generate C++, a memory-unsafe language where a single overlooked violation can become an exploitable bug. Yet most security evaluations of AI-generated code rely on static analysis alone, which flags warnings without confirming runtime violations or reasoning about untested paths. We ask whether AI-generated C++ is measurably less safe than human-written code, and whether common verification tools agree on the risk. We introduce VULBENCH-CPP, a benchmark of 8,918 C++ programs from three open-weight LLMs (Gemma 3 27B IT, LLaMA 3.3 70B Instruct, Qwen 2.5 Coder 32B Instruct) and human authors across 851 competitive-programming tasks. Each program is annotated by four verification tiers: functional testing, static analysis (cppcheck, clang-tidy), dynamic analysis (ASan/UBSan), and bounded model checking (ESBMC). Accounting for the correlation among solutions to a shared task, we find that AI-generated code is roughly twice as likely as human code to trigger a confirmed runtime violation, even after controlling for code length and test pass-rate. Under static analysis the two look equally safe, but this is misleading: the apparent similarity reflects code length rather than real safety, and the tiers detect largely different classes of violation, so no single tier is sufficient. The gap is consistent across independent generations.
