Search papers, labs, and topics across Lattice.
This paper investigates the ability of LLM agents to autonomously create cost-effective, task-specific solutions from general capabilities, a process termed "bottling." Through the BOTTLED benchmark, agents are tasked with completing unlabelled workloads within fixed resource constraints, exploring strategies like small model training or program generation. Results indicate that strong zero-shot performance does not guarantee effective bottling, with many agents underperforming baselines and even their own zero-shot capabilities, though significant cost savings are achievable for specific tasks.
LLM agents struggle to translate general capabilities into cost-efficient, reusable solutions, often failing to outperform simple distillation baselines despite significant zero-shot prowess.
Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously create cheaper solutions for such workloads? We call this ability"bottling": the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost. We introduce BOTTLED, a benchmark in which agents receive an entire unlabelled workload and must complete it under fixed time, compute and LLM API budgets. Agents choose their own approach, such as training a small model or writing a reusable program. Across ten models and three tasks, we find that strong zero-shot task performance does not reliably translate into strong bottling capabilities. Models with similar zero-shot scores can differ substantially after bottling, and 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance. Moreover, 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget. Nevertheless, bottling can yield substantial savings: on query-product relevance classification, Opus 5 retains about 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost. Bottling is also competitive with Jev, a"system one"model built especially for cheap, repetitive inference: Opus 5 on the same task recovers about 94% of Jev's macro-F1 at a quarter of Jev's projected full-workload cost. BOTTLED provides a basis for evaluating and improving agents'ability to invest limited resources in reusable solutions for large, repetitive workloads.
