Search papers, labs, and topics across Lattice.
This study investigates the impact of structured and security-oriented prompts on the generation of Python code by LLMs, specifically GPT-4o and LLaMA 3.1-8B, across 424 security-sensitive tasks. While structured prompting significantly reduces the rate of invalid outputs, it does not consistently decrease the prevalence of security weaknesses; instead, it tends to redistribute risk by lowering high-severity issues while increasing low-severity ones. The findings highlight that although prompt structure enhances compliance, it cannot replace the need for robust security measures in LLM-assisted software development.
Security-oriented prompts may reduce invalid outputs but paradoxically increase the prevalence of low-severity vulnerabilities in LLM-generated code.
Large Language Models (LLMs) increasingly generate code from natural-language prompts, making prompt engineering a key mechanism for shaping the security of generated software. Structured and security-oriented prompts are widely used to encourage safer code, yet their effects extend beyond whether detected weaknesses are simply present or absent. Using 424 security-sensitive Python tasks, we generate solutions with GPT-4o and LLaMA 3.1-8B under five prompt variants that progressively add structural and security guidance, and evaluate them with Bandit and CodeQL along two axes: generation compliance and security weakness prevalence, severity, and CWE distributions. Structured prompting substantially reduces refusals (e.g., GPT-4o invalid outputs drop from 338 of 424 to 37-52), enabling large-scale analysis, but security-oriented refinements do not consistently reduce overall weakness prevalence. For GPT-4o, stronger prompts primarily redistribute risk: high-severity findings fall (20.8% to 13.6%) while low-severity findings rise (32% to 43.5%); LLaMA shows weaker, less consistent shifts. We also observe security-driven semantic drift, where stricter prompts silently remove or rewrite explicitly requested unsafe constructs. Overall, prompt structure improves compliance but is an unreliable substitute for robust security controls in LLM-assisted development.
