Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of Google's Gemini AI tool (free vs. paid versions) on software security in a controlled experiment with 159 developers. The experiment measured code security outcomes across groups using no AI, free Gemini, or paid Gemini while controlling for developer experience. The key finding is that developer programming experience significantly improves code security, whereas no significant differences were observed between the different Gemini conditions.
Despite the promise of AI-powered tools, developer experience still trumps AI assistance when it comes to writing secure code.
The ongoing shortage of skilled developers, particularly in security-critical software development, has led organizations to increasingly adopt AI-powered development tools to boost productivity and reduce reliance on limited human expertise. These tools, often based on large language models, aim to automate routine tasks and make secure software development more accessible and efficient. However, it remains unclear how developers'general programming and security-specific experience, and the type of AI tool used (free vs. paid) affect the security of the resulting software. Therefore, we conducted a quantitative programming study with software developers (n=159) exploring the impact of Google's AI tool Gemini on code security. Participants were assigned a security-related programming task using either no AI tools, the free version, or the paid version of Gemini. While we did not observe significant differences between using Gemini in terms of secure software development, programming experience significantly improved code security and cannot be fully substituted by Gemini.