Search papers, labs, and topics across Lattice.
This paper introduces BackendForge, a benchmark designed to evaluate the capability of large language models (LLMs) in generating deployable and behaviorally correct backend services through agentic coding. By leveraging a controlled environment with contract-defined tasks and deterministic evaluation against OpenAPI contracts, the study reveals that while the top-performing model, GPT-5.5, achieves a 55.4% success rate on initial tasks, its performance drops to 28.6% when subjected to a more rigorous final oracle. These findings highlight the limitations of current LLMs in producing fully functional backend services despite their ability to handle local API behaviors.
LLMs can generate backend services that pass initial tests, but struggle significantly under comprehensive evaluation, revealing a critical gap in their capabilities.
Large language models (LLMs) are increasingly used in agentic coding settings, where they can inspect files, execute commands, run tests, observe failures, and iteratively revise code. This shift raises a central evaluation question: can an agentic LLM generate an end-to-end software artifact that is both deployable and behaviorally correct under execution? Backend services provide a controlled but realistic substrate for this evaluation. Their APIs expose application-level executable semantics, and deployed behavior can be checked deterministically against an OpenAPI contract through black-box HTTP interactions. We introduce BackendForge, a benchmark of 56 contract-defined backend generation tasks rewritten from real open-source applications. Given a visible specification and an OpenAPI contract, an LLM must generate a Dockerized service that is built, deployed, and evaluated only through HTTP tests. To strengthen evaluation without introducing hidden requirements, BackendForge uses a test agent and a code agent to co-evolve the test oracle and reference service, where the test agent proposes specification-grounded backend tests and the code agent repairs the reference implementation. Although the best-performing model, GPT-5.5, succeeds on 55.4\% of tasks under the base oracle, it succeeds on only 28.6\% under the final oracle. This gap suggests that current LLMs can implement many local API behaviors, but still struggle to produce complete backend services.
