Search papers, labs, and topics across Lattice.
This paper introduces CodeChat-Eval, an evaluation framework designed to assess the functional correctness of Large Language Models (LLMs) during multi-turn code refinement dialogues, a scenario often overlooked in existing benchmarks. The study reveals a significant decline in functional correctness, with reductions ranging from 19.2% to 69.2% across different models, particularly when logic-level refinements and additive changes are requested. These results underscore the challenges LLMs face in maintaining code functionality through iterative refinement, emphasizing the necessity for improved evaluation metrics in this context.
LLMs can lose up to 69% of functional correctness during multi-turn code refinements, revealing critical gaps in their reliability for software engineering tasks.
Large Language Models (LLMs) are increasingly used in software engineering to generate and refine code. In practice, developers often continue from an initial code generation request with follow-up refinement instructions, such as requests to improve style, restructure implementation, or change the execution strategy while preserving the intended behaviour. However, existing benchmarks generally omit this multi-turn code refinement dialogue setting and therefore cannot evaluate whether LLMs maintain functional correctness, i.e., whether the refined code still passes the test suite for the original task. To address this limitation, we introduce CodeChat-Eval, an evaluation framework that constructs evaluation sessions from multi-turn code refinement dialogues using a dynamic instruction selection algorithm. Our empirical study on open-weight and proprietary LLMs observes a statistically significant decrease ranging from 19.2% (GPT-5 Nano) to 69.2% (Llama 3.1 8B) in functional correctness over multi-turn refinement. The largest correctness drops are associated with logic-level refinements and additive change requests. These findings indicate that LLMs struggle to maintain functional correctness during multi-turn code refinement dialogues, and highlight the need for benchmarks that evaluate functionality-preserving refinement beyond single-turn generation.
