Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of unit tests generated by Large Language Models (LLMs) in relation to the maintainability of source code, as quantified by CodeScene's CodeHealth (CH) metrics. By analyzing test performance across Python, Java, and C++ using coverage metrics and mutation scores, the authors find that while CH is a weak predictor of test effectiveness, it consistently correlates with a reduction in input-token count. These insights highlight the nuanced relationship between code maintainability and the efficiency of LLM-generated tests, suggesting implications for software engineering practices.
LLM-generated tests are less effective on poorly maintained code, revealing a surprising link between code health and token efficiency.
Coding agents powered by Large Language Models (LLMs) are now prominent in software engineering. Previous work has shown that AI tools perform better on high-quality source code that is easy to maintain. In this study, we investigate how the effectiveness of LLM-generated unit tests varies across maintainability levels measured by CodeScene's CodeHealth (CH). We assess test effectiveness using traditional coverage metrics and mutation score across Python, Java, and C++. Moreover, we study how code with different levels of CH translates into input tokens using common industrial tokenizers. Our results suggest that CH provides a weak but consistent signal of LLM-generated test effectiveness and is negatively correlated with input-token count. These findings provide further evidence for a relationship between maintainability and LLM-based software development.
