Search papers, labs, and topics across Lattice.
This study evaluates the security compliance of Ansible code generated by 16 AI models, revealing that all models produced code with security vulnerabilities when not guided by security standards. By integrating Ansible best practices and CIS benchmarks into the prompt design using an extended CO-STAR framework, the authors demonstrate that 4 out of the 16 models can generate compliant code, achieving up to 100% CIS compliance. This approach not only significantly outperforms human-generated code but also enhances overall code quality by 19%-49%, highlighting the potential for proactive security measures in AI-assisted coding.
AI-generated code can be made secure and compliant, with some models achieving up to 100% compliance when guided by best practices.
AI coding assistants generate Infrastructure as Code, yet no work has examined whether this code meets security requirements. This matters because security smells in infrastructure code propagate to deployed systems, producing infrastructure that is insecure and untrustworthy. We evaluate 16 AI models generating Ansible roles for Apache Tomcat v10 and MongoDB v7, analysing 278 Ansible roles against CIS benchmarks. Without security guidance, all 16 AI models produced code containing security smells, resulting in vulnerable infrastructure that fails compliance verification and underperforms code written by human developers. We introduce an approach integrating Ansible best practices and CIS benchmarks into prompts through an extended CO-STAR framework, enabling security smell prevention during synthesis rather than detection after deployment. When this approach is applied, 4 out of 16 models generate compliant code, with the leading model achieving 95%-100% CIS compliance, a fourfold improvement over humans at 23%-43%, with overall code quality improving by 19%-49%. The remaining 12 models fail not because they cannot generate code but because they cannot follow instructions with multiple constraints. For capable models, the approach requires no retraining and can be adopted through system prompts.
