Search papers, labs, and topics across Lattice.
EcoAgent-Bench introduces a novel benchmark for evaluating economic decision-making in budget-constrained LLM agents, emphasizing the importance of resource allocation in task completion. The benchmark comprises 304 tasks derived from real-world scenarios, assessing agents on their ability to make economically sound decisions, such as when to escalate tasks or select model tiers. Results reveal that while some agents achieve high accuracy, they often fail to demonstrate economic consistency, highlighting a critical gap between task completion and resource-efficient decision-making.
Economic decision-making in LLM agents reveals a stark divide between task completion and resource efficiency, with agents often overspending or under-escalating.
Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic. In deployment, however, the choice among a local lookup, broad search, composite research tool, stronger model, or human escalation is part of the task itself. We introduce EcoAgent-Bench, in which every task specifies priced actions and an explicit budget. Its 304 real-derived tasks span five families adapted from GAIA, HotpotQA, and MuSiQue, and test four decisions: avoiding unnecessary escalation, escalating when local evidence is insufficient, selecting a model tier, and stopping on unsupported premises. We evaluate seven LLM agents in tool-API and workspace-CLI settings, together with four oracle scripted controls. Micro-averaged accuracy rewards one-sided policies: always-escalate controls achieve high micro success while failing save-oriented tasks. We therefore also report an economic-consistency score (the worse of accuracy on upgrade-oriented and save-oriented family groups) which exposes this failure. Tool-API agents attain only 3.9-24.0% micro strict success (at most 7.3% economic consistency), often either stopping before warranted escalation or overspending on cheap tasks. A threshold-crossing budget sweep changes GPT-5.4's escalation rate from 0% to only 3%. These results show that completion under a budget and economical action selection are distinct properties. We release the task bundle, transformation pipeline, frozen evaluation environments, and integrity-bound result artifacts needed to study both.
