Search papers, labs, and topics across Lattice.
This paper critiques existing coding benchmarks for AI agents, highlighting their misalignment with the realities of agentic software engineering. The authors argue that these benchmarks fail to differentiate between the contributions of models, harnesses, and environments, leading to misleading evaluations that obscure the iterative potential of individual components. By identifying key symptoms of this misalignment, the study calls for a reevaluation of how coding agents are assessed to better reflect their operational complexities and foster meaningful improvements.
Current coding benchmarks obscure the true performance of AI agents by conflating models with their operational contexts, stifling innovation in software engineering.
Coding agents have become a major mode of software engineering, but the benchmarks we use to compare them were designed in a pre-agent era: they collapse model, harness, and environment into a single end-to-end score, typically computed against one reference solution, with no component-level signal for iteration. We argue that current coding benchmarks are misaligned with agentic software engineering. A coding agent in practice is not a model: it is a system harness -- a composite of models, harnesses, contexts, environments, and feedback signals, any one of which can move the benchmark score by margins comparable to those between adjacent model generations. We discuss three symptoms: (i) benchmark scores conflate the model with the rest of the harness; (ii) grading against a single reference solution penalises equally valid alternatives; and (iii) the absence of signal at the level of individual harness components makes the end-to-end system score difficult to iterate on.
