Search papers, labs, and topics across Lattice.
DeepSWE introduces a novel benchmark consisting of 113 original, long-horizon software engineering tasks specifically designed to evaluate coding agents without the biases present in existing benchmarks. By creating tasks from scratch across diverse open-source repositories and employing a hand-written verifier for grading, DeepSWE ensures that evaluations reflect genuine problem-solving capabilities rather than mere recall of pre-trained knowledge. The results show that DeepSWE's grading aligns significantly better with independent LLM assessments compared to traditional benchmarks, while also providing a more nuanced score distribution for frontier coding agents.
Coding agents can now be evaluated on tasks that truly test their problem-solving skills, rather than their ability to recall previously seen solutions.
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.
