Search papers, labs, and topics across Lattice.
This paper introduces Leyline, a novel serving-side primitive designed to address the unique challenges posed by agentic LLMs, which require dynamic cache management due to their policy-driven conversation evolution. Leyline enables precise cache editing by separating the directives for content modification from the preservation of position correctness, allowing for efficient in-place splicing and semantic forgetting without the need for full prefix recomputation. Experimental results show that Leyline significantly improves cache hit rates by 11.2 percentage points and reduces latency by up to 241 ms, while also enhancing the agentic solve rate by 14.3 percentage points in a debug-gym environment.
Leyline transforms agentic LLM performance by enabling efficient, policy-driven cache edits that drastically reduce latency and improve solve rates.
Modern KV cache management assumes the chatbot workload: prompts arrive once and the cache grows append-only, so prefix caching and forward-only eviction are correct by construction. Agentic LLMs break this assumption. Their conversations evolve through policy-driven editing: failed tool calls are retried, stale outputs dropped, trajectories pivoted. Two distinct cache problems result. First, identical content moves to new positions between turns, invalidating exact-prefix caches even though the underlying KV would still be valid; recent work on position-independent caching for MLA addresses this reuse problem. Second, and this paper's focus, a policy may need to direct the serving system to actively remove or replace a span of cached content and continue without re-prefilling everything that came after. No existing primitive offers this. Production agentic harnesses fall back to re-prefill on every edit, paying full prefix-recomputation cost; kernel-level eviction methods make their own decisions and cannot accept policy directives from outside the kernel. We introduce Leyline, a serving-side primitive that closes this gap. A declarative directive 4-tuple separates what to edit from how to preserve position correctness. The policy declares the edit and its mode (in-place splice or prefix-trimmed re-prefill for semantic forgetting); an architecture-agnostic interface routes to a per-architecture kernel that restores attention math via a closed-form RoPE-rotation correction. The splice kernel lifts replay cache-hit by +11.2 pp and cuts latency by up to 241 ms. A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 pp on debug-gym. The mechanism is open; the policy space it enables is the agenda.
