Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study conducts a three-arm ablation analysis comparing the performance of coding agents using different tool surfaces—baseline, bash-only, and code-only—across synthetic computation and SWE-bench Mini tasks. The findings reveal that restricting agents to a single execute_code tool is often as cost-effective as using multiple tools, with significant cost savings observed in specific configurations. Notably, the analysis indicates that the optimal tool surface is influenced by both the task regime and agent design, challenging the notion that one size fits all in coding agent performance.
Restricting coding agents to a single execution tool can be as effective and cheaper than using multiple tools, revealing a nuanced interplay between task and agent design.
Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.