Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper benchmarks seven tactics for reducing cloud LLM token usage in coding agent workflows by using a local model as a triage layer. They implement these tactics in an open-source shim and evaluate them individually and in combination across four coding-agent workload classes, measuring token savings, cost, latency, and routing accuracy. The key finding is that local routing and prompt compression achieve 45-79% cloud token savings on edit-heavy and explanation-heavy workloads, while a combination of tactics achieves 51% savings on RAG-heavy workloads, with the optimal tactic subset being workload-dependent.
Stop overpaying for cloud LLM tokens: a local model can slash your costs by up to 79% on coding tasks, but the best strategy depends on the workload.
We present a systematic measurement study of seven tactics for reducing cloud LLM token usage when a small local model can act as a triage layer in front of a frontier cloud model. The tactics are: (1) local routing, (2) prompt compression, (3) semantic caching, (4) local drafting with cloud review, (5) minimal-diff edits, (6) structured intent extraction, and (7) batching with vendor prompt caching. We implement all seven in an open-source shim that speaks both MCP and the OpenAI-compatible HTTP surface, supporting any local model via Ollama and any cloud model via an OpenAI-compatible endpoint. We evaluate each tactic individually, in pairs, and in a greedy-additive subset across four coding-agent workload classes (edit-heavy, explanation-heavy, general chat, RAG-heavy). We measure tokens saved, dollar cost, latency, and routing accuracy. Our headline finding is that T1 (local routing) combined with T2 (prompt compression) achieves 45-79% cloud token savings on edit-heavy and explanation-heavy workloads, while on RAG-heavy workloads the full tactic set including T4 (draft-review) achieves 51% savings. We observe that the optimal tactic subset is workload-dependent, which we believe is the most actionable finding for practitioners deploying coding agents today.