Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
Adaptive DHiSparse addresses the KV memory bottleneck in long-context LLM serving by introducing a load-adaptive hierarchical sparse KV cache that integrates with speculative decoding. This approach dynamically manages KV cache residency between GPU and host memory, fetching only necessary history and retaining more on-GPU memory when available. At 100K context and high concurrency, Adaptive DHiSparse with DFlash2 achieves 1.45x the throughput of HiSparse and 1.27x that of non-speculative adaptive serving, while also enabling significantly higher concurrency.
Speculative decoding now achieves 4x concurrency and 3.7x output throughput on 100K context LLMs by intelligently managing KV cache residency across GPU and host memory.
Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45$\times$ HiSparse and 1.27$\times$ non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4$\times$ the concurrency and 2.3 to 3.7$\times$ the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85$\times$ the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.