Search papers, labs, and topics across Lattice.
ProdCodeBench, a new benchmark for evaluating AI coding agents, is derived from real-world production sessions, addressing limitations of existing benchmarks in reflecting actual usage. The benchmark includes verbatim prompts, committed code changes, and fail-to-pass tests across seven programming languages, all extracted from a production AI coding assistant's usage. Evaluation of four foundation models on ProdCodeBench reveals solve rates between 53.2% and 72.2%, with higher solve rates observed in models leveraging work validation tools like test execution and static analysis.
Real-world coding benchmarks reveal that AI coding agents succeed more often when they iteratively validate their work with tests and static analysis, suggesting a path to better agents in unfamiliar codebases.
Benchmarks that reflect production workloads are better for evaluating AI coding agents in industrial settings, yet existing benchmarks differ from real usage in programming language distribution, prompt style and codebase structure. This paper presents a methodology for curating production-derived benchmarks, illustrated through ProdCodeBench - a benchmark built from real sessions with a production AI coding assistant. We detail our data collection and curation practices including LLM-based task classification, test relevance validation, and multi-run stability checks which address challenges in constructing reliable evaluation signals from monorepo environments. Each curated sample consists of a verbatim prompt, a committed code change and fail-to-pass tests spanning seven programming languages. Our systematic analysis of four foundation models yields solve rates from 53.2% to 72.2% revealing that models making greater use of work validation tools, such as executing tests and invoking static analysis, achieve higher solve rates. This suggests that iterative verification helps achieve effective agent behavior and that exposing codebase-specific verification mechanisms may significantly improve the performance of externally trained agents operating in unfamiliar environments. We share our methodology and lessons learned to enable other organizations to construct similar production-derived benchmarks.
Meta AI