Search papers, labs, and topics across Lattice.
This paper introduces DUALFIX, a novel search-based approach for optimizing natural language prompts to enhance the code generation performance of large language models (LLMs). By evolving transformation rules that are error-agnostic and reusable, DUALFIX addresses both specification and implementation failures in code generation, achieving significant improvements in fixing failing cases. The method demonstrates a recovery rate of up to 30% of baseline failures and outperforms traditional execution-feedback repair methods, showcasing its generalizability across different models and benchmarks.
Evolved prompt transformation rules can fix 10-30% of LLM code generation failures, outperforming traditional execution-feedback methods.
Large language models are known to be sensitive to prompt formulation. Even minor variations in wording can substantially degrade performance. This sensitivity reveals an opportunity: if prompt phrasing can harm performance, can it be used to improve it? To investigate this question, we introduce a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance. We then propose DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures. A key strength of our approach lies in its generality: the evolved rules are error-agnostic, reusable across problems, and transferable across models. We evaluate DUALFIX against execution-feedback repair baselines across three models on two challenging benchmarks, LiveCodeBench and APPS. Our results show that the evolved transformations fix from 10-30% of failing cases, including 12-17% of failures that execution-based repair alone cannot resolve. Overall, DualFix recovers up to 30% of baseline failures and fixes 3-5 times more failing cases than Self-Fix across all evaluated settings. Furthermore, we also show that rules evolved on one model transfer zero-shot to other models, outperforming execution-feedback repair without any re-optimization.
