Search papers, labs, and topics across Lattice.
This study systematically investigates the iterative refinement capabilities of Large Language Models (LLMs) in code correction by utilizing execution feedback from compiler errors and test cases. By evaluating four models across two programming languages, the research introduces new metrics for assessing code failures and rectification patterns, revealing that reasoning models significantly outperform non-reasoning models in leveraging feedback. Key findings indicate that while syntactic and runtime errors are more easily corrected, logical and algorithmic failures present greater challenges, highlighting the importance of feedback loops in LLM-driven code generation systems.
Reasoning models in LLMs can dramatically improve code correction accuracy through iterative feedback, outperforming their non-reasoning counterparts.
Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs' ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.
