Search papers, labs, and topics across Lattice.
This paper introduces a comprehensive evaluation framework for assessing the quality of code generated by Large Language Models (LLMs) in the .NET ecosystem, focusing on functional correctness, maintainability, and efficiency. By conducting a controlled experiment on 340 solutions across 85 algorithmic tasks, the study reveals a significant disconnect between traditional Pass@k metrics and the actual performance of LLMs, highlighting a Pearson correlation of only 0.075 between correctness and quality attributes. The findings underscore the inadequacy of existing benchmarks, particularly in enterprise contexts, and detail GPT's unique bimodal failure behavior in code generation.
LLMs exhibit a striking disconnect between functional correctness and code quality, with traditional metrics failing to capture the full spectrum of performance.
Evaluating Large Language Model (LLM) code generation quality requires examining not just whether the generated code is correct, but whether it is maintainable, efficient, and stylistically sound, all of which are qualities of direct importance to software engineering practitioners. Existing benchmarks reduce evaluation to a single Pass@k metric, which obscures critical trade-offs between functional correctness and structural quality. A further limitation is the near-exclusive focus on Python, leaving enterprise-relevant ecosystems such as C# and .NET without dedicated evaluation. This paper presents an automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok. We conduct a controlled experiment across 85 algorithmic tasks derived from HumanEval, generating and evaluating 340 solutions in total, in which each solution is assessed across three independent dimensions: functional correctness via automated unit testing, static code quality via Roslyn AST analysis, and runtime efficiency via adversarial BenchmarkDotNet profiling. Our central finding reveals a substantial gap between correctness and quality attributes (Pearson r = 0.075), demonstrating that Pass@k rankings systematically misrepresent the full LLM performance profile in software engineering contexts. We further characterize GPT's bimodal failure behavior.
