Search papers, labs, and topics across Lattice.
This paper introduces KernelFlume, an innovative architecture designed to optimize long-context decoding for large language models (LLMs) by disaggregating the stable projection and feedforward neural network (FFN) paths from core-attention computation. By employing a routing table that dynamically maps token ranges to attention-node endpoints, KernelFlume achieves significant reductions in both latency and operational costs compared to traditional instance-level scaling methods. The results demonstrate that KernelFlume can sustain low per-token latencies while reducing costs by up to 61% on real GPU testbeds, making it a compelling solution for handling bursty long-context demands in LLM serving.
KernelFlume slashes operational costs by up to 61% while maintaining low latency for long-context LLM decoding, revolutionizing how we scale model serving.
LLM serving is increasingly dominated by long and dynamic decode workloads from agents, reasoning models, and extended conversations. When bursty long-context demand exceeds deployed capacity, existing serving systems typically scale out by launching additional serving instances with model replicas. This instance-level elasticity increases KV capacity only by provisioning another full copy of the model, inheriting startup latency, memory overhead, and batch fragmentation. We present KernelFlume, a decode-centric architecture that disaggregates the stable projection/FFN path from core-attention computation: weight nodes execute dense projection/FFN kernels, while weightless attention nodes store token-range KV partitions and scale with request-state demand. To make this separation elastic, KernelFlume maintains a routing table that maps token ranges to attention-node endpoints. It updates routes at token boundaries and uses host-visible graph signals to drive pre-registered UCX endpoint communication outside the captured CUDA Graph. To preserve low per-token latency after disaggregation, KernelFlume combines query-first core-attention dispatch with inter-layer kernel pipelining, overlapping remote attention and communication with local projection/FFN work. On real GPU testbeds (intra-node A6000 and cross-node H100), under a dynamic long-context agentic workload serving Llama-3.1-8B, KernelFlume sustains flat p99 TPOTs of ~74 ms on A6000 and ~34 ms on H100, while lowering cost per million output tokens by up to 32% and 61%, respectively, relative to full-instance elastic scaling with ServerlessLLM, a state-of-the-art instance-startup method. Replaying the same trace at larger model scale in simulation projects a 56--66% cost reduction over ServerlessLLM, widening to 80--85% with cheaper heterogeneous attention-node hardware and persisting into the million-token context range.
