Search papers, labs, and topics across Lattice.
This study investigates the security calibration of large language models (LLMs) in generating code, assessing whether these models can accurately gauge the security of their outputs. Evaluating GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next across various benchmarks, the authors find that these models exhibit a concerning level of overconfidence, particularly in functional calibration compared to security calibration. The research highlights the challenges of using calibration-guided automated repair, which shows limited effectiveness and often leads to functional regressions, emphasizing the need for improved calibration strategies in real-world applications.
Overconfidence in LLM-generated code is rampant, with models often misjudging the security of their outputs, raising significant concerns for software safety.
Large Language Models (LLMs) are rapidly transforming software development, yet their use in security-critical contexts raises a key question: do models know when their generated code is insecure? This property, known as calibration, measures whether a model's confidence aligns with the true correctness of its outputs. We present the first large-scale empirical study of security calibration in LLM-generated code. We evaluate GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next across multiple temperature settings on two complementary benchmarks: self-contained security tasks and multi-language repository-level contexts. Our results suggest that overconfidence is prevalent across the evaluated LLMs. Functional calibration is consistently worse than security calibration, suggesting that models estimate security outcomes more reliably than functional correctness, potentially because functional correctness depends on complex execution behavior. We also examine whether calibration-guided automated repair can help remediate vulnerabilities in LLM-generated code, finding only limited improvements while frequently introducing functional regressions. Moreover, we study different mitigation strategies for reducing False Trust, where models assign high confidence to vulnerable code. The results show that although architectural gating improves calibration on controlled benchmarks, calibration deteriorates in realistic repository-level settings, increasing the risk of high-confidence vulnerable outputs.
