Search papers, labs, and topics across Lattice.
To evaluate whether coding agents can deliver production-ready systems under messy client constraints, the authors develop $\tau^\tau$-bench, a 53-task benchmark spanning four domains where developer agents must build deployable customer-service agents from real-world business records, APIs, and cost budgets. Unlike static code synthesis benchmarks, the synthesized agents are evaluated end-to-end via live execution against held-out interactive user simulations. Frontier agents struggle severely with this meta-engineering task: Claude Opus 5 driven by Claude Code achieves just a 23.9% pass rate against an 82.2% expert-authored ceiling, routinely shipping brittle first-draft architectures without exploring design trade-offs or clarifying client intent.
Even frontier coding agents collapse to a 23.9% success rate when tasked with end-to-end agent construction under realistic client constraints, falling nearly 60 percentage points short of human engineers.
LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce $\tau^\tau$-bench (pronounced hyper-tau-bench), a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for $\tau^\tau$-bench to turn the work of cooperative agent building into a measurable target for coding agents.
