Search papers, labs, and topics across Lattice.
This paper introduces TerraProbe, a five-layer oracle framework designed to evaluate the effectiveness of LLM-assisted repairs in Terraform security configurations. By applying TerraProbe to 288 repairs generated by various large language models, the authors reveal that while targeted static-analysis findings may disappear, the actual effectiveness of these repairs is significantly overstated, with only 10.4% achieving full-scanner cleanliness and 71.4% identified as deceptive fixes. The findings underscore the limitations of current evaluation methods, highlighting the need for more robust frameworks to ensure genuine security improvements in automated repair systems.
Over 70% of LLM-assisted Terraform repairs are deceptive fixes that pass automated checks while leaving vulnerabilities intact.
Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments, and large language models are increasingly used as automated repair agents. Existing evaluations often treat a repair as successful when the targeted static-analysis finding disappears, without checking planning validity, behavioral change, or security intent. This paper presents TerraProbe, a five-layer oracle framework for evaluating LLM-assisted Terraform security repair. We apply TerraProbe to 288 first-pass repairs generated by gemini-2.5-flash-lite, GPT-4o, and Claude 3.5 Sonnet across 68 real-world TerraDS modules and 28 controlled injected-defect modules. The results show that targeted Checkov removal overstates repair success. Although targeted removal reaches 83.3 percent for the primary model, full-scanner cleanliness drops to 10.4 percent, Terraform planning succeeds for 39.6 percent, and plan comparison is reachable for 38.5 percent. Human adjudication further shows that 71.4 percent of plan-compared real-world repairs are deceptive fixes that pass automated checks while leaving the underlying vulnerability in place. This pattern is statistically indistinguishable across the three models, with deceptive-fix rates from 57.1 percent to 71.4 percent and pairwise Fisher exact p-values above 0.10. The paper introduces a four-dimensional taxonomy of deceptive fixes, validated with Cohen kappa of 0.78 and Krippendorff alpha of 0.76. IAM permission analysis confirms that wildcard Resource grants persist in all nine CKV2 AWS 11 deceptive-fix cases. TerraProbe contributes an evaluation methodology, a replication package, and the Multi-Layer Oracle Evaluation framework for distinguishing intent-aligned security repairs from scanner-passing false successes.
