Search papers, labs, and topics across Lattice.
This study analyzes the security vulnerabilities present in code generated by three prominent LLM-based coding tools: ChatGPT, Microsoft Copilot, and Google Gemini. By conducting a standardized vulnerability review using the CVSS v3.1 scoring system and mapping findings to the OWASP Top 10:2021 and MITRE ATT&CK frameworks, the researchers found that all generated scripts contained exploitable vulnerabilities, with a significant overlap in vulnerability classes across models. The results highlight that the risk of deploying LLM-generated code lies not in the specific model used, but in the inherent vulnerabilities associated with automation tasks, emphasizing the need for security reviews before deployment.
Every LLM-generated automation script analyzed contained exploitable vulnerabilities, regardless of the model used.
LLM-based coding tools enable non-expert users to generate routine automation scripts that may enter enterprise workflows without meaningful security review. This study examines that risk directly. Code was collected from ChatGPT, Microsoft Copilot, and Google Gemini using identical prompts across three automation domains. Claude Code performed a standardized vulnerability review. Each identified vulnerability was scored using CVSS v3.1 and mapped to the OWASP Top 10:2021 and the MITRE ATT&CK frameworks. Every script contained exploitable vulnerabilities. Nine of the 17 identified vulnerability classes appeared in code from all three models, while 14 of the 17 vulnerability classes appeared in at least two models. The weighted CVSS scores across platforms differed by less than 10%. The risk is not tied to any particular model but rather to the task category. Organizations should therefore ask not which tool to trust, but instead whether LLM-generated automation code should be deployed without review.
