Search papers, labs, and topics across Lattice.
This paper investigates the economic implications of caching in large language model (LLM) workloads, particularly focusing on agentic tasks that disrupt the efficiency of cached prompts. By implementing a client-side keepalive mechanism, the authors demonstrate that it can significantly reduce the cost of follow-up requests by maintaining the cache during idle periods, achieving cost reductions of up to 12.5 times. The findings suggest that optimizing the ping frequency for keepalive can yield substantial savings, with implications for how LLM providers might adjust their caching policies in response to user behavior.
A client-side keepalive can slash follow-up request costs by up to 12.5x, reshaping the economics of LLM usage in agentic workloads.
Frontier LLM providers cache a prompt's processed prefix so that a follow-up request sharing it pays ~10% of the input price and skips most of the prefill latency. Agentic workloads systematically destroy this benefit: the agent sends a request, runs a tool or waits for approval for minutes, and by the time the follow-up is sent the cached prefix has been evicted, so the agent pays the full prefill again. A client-side keepalive, replaying the prefix on a timer during the pause, prevents this, and it is individually rational: across Anthropic, OpenAI, Google, and DeepSeek we show that a keepalive holds the prefix warm through gaps where idle baselines are evicted, cutting the post-pause request cost by up to 12.5x. The strategic question is the ping frequency, and it has a clean answer: keepalive cost falls monotonically in the interval, so the economical choice is the largest interval safely under the provider's TTL, about 4 minutes at Anthropic's 5-minute TTL rather than the 30-second convention, and the strategy breaks even against a re-prefill at idle ~tau(w/r - 1) (~46 min for Anthropic, ~36 min for OpenAI and DeepSeek). Because the benefit is real and bounded only by each user's own bill, rational adoption is universal adoption; and since cache residency is priced per read rather than per token-hour, a keepalive-saturated tier gives LRU eviction nothing to rank. We argue this externality will push providers to meter cache residency directly, and one already does. We derive the operator's policy until then.
