Search papers, labs, and topics across Lattice.
This paper critically evaluates the effectiveness of existing function-level performance benchmarks for large language models (LLMs) by rigorously testing 1,538 tasks and revealing that only 6.11% of LLM-generated implementations significantly outperform canonical solutions. A manual analysis of non-significant tasks uncovers that many performant implementations either show no meaningful change or have undiscovered potential improvements. To enhance performance evaluation, the authors introduce an LLM-based multi-agent framework that generates more effective performance-oriented tests, achieving statistically significant improvements in over 24% of previously non-significant tasks, surpassing current state-of-the-art methods.
Existing benchmarks fail to reveal the true performance capabilities of LLMs, with only 6.11% showing significant speed advantages over traditional implementations.
Many function-level performance benchmarks have been proposed to evaluate whether large language models (LLMs) can generate efficient programs. However, results on these benchmarks often show that LLM-generated implementations have little or no execution-time difference from canonical solutions. In this paper, we revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury. We evaluate 1,538 tasks under more rigorous setting by running each task 30 times and assessing the runtime differences between the canonical solutions and benchmark-provided performant implementations with statistical testing. With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions. In a manual analysis of 308 non-significant tasks, 99 performant implementations contain no meaningful performance change, while 209 contain potential performance improvements that are not exposed by the original tests. These results suggest that the main limitation is not only the evaluation method, but also the limited sufficiency of the benchmark-provided performance tests. To address this limitation, we propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests. The framework uses three separate agents to generate, diagnose, and repair deterministic tests that preserve functional correctness while better exposing performance differences. Across 1,345 benchmark tasks for which the original tests found no significant performance difference, tests generated by our framework with DeepSeek-v3.1 and GPT-4o reveal statistically significant improvements in 24.01% and 25.43% of the tasks, respectively, outperforming the SOTA LLM-based performance test generation method.
