Search papers, labs, and topics across Lattice.
This paper introduces LLM-as-a-Developer, a novel methodology that employs a large language model to autonomously code and validate agents using various Agent Development Kits (ADKs). By systematically evaluating 51 popular Python ADK frameworks, the study reveals significant variations in API usability and agent performance, with generation success rates reaching 57% and costs varying by 5.6 times across frameworks. Notably, while no single framework consistently outperforms others, some ADKs can produce agents that outperform general-purpose coding agents at a lower cost, challenging assumptions about framework superiority.
Agent Development Kits vary dramatically in usability, with some enabling agents to outperform general-purpose coding tools at a fraction of the cost.
The rapid proliferation of Agent Development Kits (ADKs), SDK-level frameworks for building LLM-powered autonomous agents, has outpaced any empirical understanding of how framework choice affects agent performance. We propose \textbf{LLM-as-a-Developer}, a methodology that replaces human developers with an LLM coding agent that learns each framework's API from documentation, writes agent code, and iteratively repairs it through a validate-and-feedback loop until tests pass. By holding the developer constant and varying only the framework, generation effort becomes a quantitative proxy for API usability and the resulting agents provide a controlled measure of framework effectiveness. We implement this in \textbf{ADK Arena}, a fully automated pipeline with per-framework Docker isolation, a three-level validation pipeline, and benchmark adapters for SWE-bench, $τ^2$-bench, Terminal-Bench, and MCP-Atlas. Evaluating all 51 popular Python ADK frameworks (204 agent--benchmark pairs), we find that: (1)~generation succeeds for 57\% of runs, and its cost varies 5.6$\times$ across frameworks (\$0.6 to \$3.4 per agent), a quantitative proxy for API complexity, though cost alone does not predict success; (2)~no single framework dominates: the best single-benchmark ADK agents resolve up to 80\% of tasks and can even \emph{beat} general-purpose frontier coding agents at a fraction of the cost, yet the median framework resolves only 32\%; (3)~across information-source ablations, genuine framework usage stays within a narrow 28--40\% band (highest with raw source access and still 33\% with no reference material at all), indicating that documentation, source code, and parametric knowledge are largely substitutable rather than any one being a hard bottleneck.
Microsoft Research