Search papers, labs, and topics across Lattice.
SparseEngine addresses the memory and computational bottlenecks of long-context LLM agents by introducing a sparse-first inference engine designed for heterogeneous sparse attention methods. Its novel shared lifecycle contract and Chain Cache mechanism allow diverse KV representations and cross-request state management, enabling efficient KV eviction and pruning. This results in significant performance gains, including over 10x higher throughput with KV eviction and over 2x end-to-end speedup on agent benchmarks, while preserving method quality.
A single inference engine now unifies 15 sparse attention methods, unlocking over 10x throughput gains for long-context LLMs through intelligent KV cache management.
Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.
