Search papers, labs, and topics across Lattice.
This study critically examines the non-functional quality of code patches generated by later models compared to earlier ones, focusing on repository-level repair tasks. By conducting case studies with Claude and DeepSeek models, the researchers found that while later models resolved more instances, they did not consistently improve on non-functional metrics such as CPU time and peak memory usage. The findings suggest that relying solely on resolved rates may overlook important aspects of model performance in practical software engineering contexts.
Later models may resolve more coding tasks, but they don't necessarily produce better quality patches in terms of performance metrics.
Repository-level coding benchmarks typically measure progress in model capability by comparing the resolved rates of later and earlier models. However, this focus overlooks whether the non-functional quality of their generated patches has also changed across model generations. This study investigates whether later models produce functionally correct patches with better non-functional characteristics than earlier models on comparable repository-level repair tasks. We conducted two case studies involving four Claude and DeepSeek models on SWE-bench Lite. Using the same SWE-agent functional repair setting, we evaluated the generated patches with CodeQL, CodeScene, CPU time, and peak memory. Our primary analysis compared the models on commonly resolved instances. The static analysis results showed that most CodeQL paired differences were zero and that no CodeQL or CodeScene comparison remained significant after Holm correction. CPU time differences were small and inconsistent across model families, while peak memory usage was slightly higher for the later models under the benchmark test workload, with small absolute differences. Differences in individual CodeQL rules and CodeScene categories varied across model families and did not survive multiple-comparison correction. Overall, later models resolved more instances but showed no consistent improvement in the measured non-functional indicators on tasks solved by both models. Through this study, we hope to encourage a more comprehensive evaluation of models'practical software engineering capabilities.
