Search papers, labs, and topics across Lattice.
This study empirically evaluates the effectiveness of iterative repair loops in LLM-based software engineering tasks, revealing that the initial three to four iterations yield the majority of performance improvements. The research highlights that subsequent iterations offer diminishing returns, suggesting that the orchestration of workflows and the design of feedback mechanisms play a more critical role than the choice of the underlying model. By treating repair budgets as a variable, the findings have significant implications for evaluation outcomes, computational efficiency, and reproducibility in the field.
Most gains in LLM-based repair loops come from just the first three iterations, challenging the assumption that more iterations always lead to better results.
Iterative repair loops have become a core design pattern in LLM-based software engineering systems. These workflows repeatedly generate, validate, and repair artifacts using feedback such as compiler errors or test failures. Despite their widespread use, the impact of repair-loop iteration limits remains poorly understood, as most prior work adopts fixed, often arbitrary, repair budgets. We study repair-loop effectiveness across multiple software engineering tasks, including code generation, test generation, and code translation. Across several representative workflows, datasets, and contemporary low-cost LLMs, we observe a consistent pattern of diminishing returns: the first three to four repair iterations account for most achievable gains, while later iterations contribute only marginal improvements. We further find that repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself. These results suggest that repair budgets should be treated as an explicit experimental variable, as they directly affect evaluation outcomes, computational cost, runtime, and reproducibility in LLM-based software engineering research.
