Search papers, labs, and topics across Lattice.
The paper introduces CodeTaste, a benchmark for evaluating LLMs' ability to perform code refactorings that align with human developer choices in open-source projects. It assesses LLMs on both executing specified refactorings and identifying appropriate refactorings given a general area for improvement, using repository test suites and static checks for scoring. Results show that while LLMs can execute refactorings when explicitly instructed, they struggle to independently identify the refactorings that humans would choose, but a propose-then-implement strategy improves alignment.
LLMs can follow detailed code refactoring instructions, but still fall short of mimicking human refactoring choices in real-world codebases, highlighting a critical gap in their ability to autonomously improve code quality.
Large language model (LLM) coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refactoring: behavior-preserving program transformations that improve structure and maintainability. In this paper, we investigate if LLM agents (i) can execute refactorings reliably and (ii) identify the refactorings that human developers actually chose in real codebases. We present CodeTaste, a benchmark of refactoring tasks mined from large-scale multi-file changes in open-source repositories. To score solutions, we combine repository test suites with custom static checks that verify removal of undesired patterns and introduction of desired patterns using dataflow reasoning. Our experimental results indicate a clear gap across frontier models: agents perform well when refactorings are specified in detail, but often fail to discover the human refactoring choices when only presented with a focus area for improvement. A propose-then-implement decomposition improves alignment, and selecting the best-aligned proposal before implementation can yield further gains. CodeTaste provides an evaluation target and a potential preference signal for aligning coding agents with human refactoring decisions in realistic codebases.
ETH