Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces NetKV, a novel scheduling approach for disaggregated LLM inference that incorporates network-aware decoding by utilizing a network cost oracle to optimize the selection of decode instances. By addressing the limitations of existing schedulers that overlook network topology and congestion, NetKV achieves significant reductions in Time to First Token (TTFT) and improves service level objective (SLO) attainment. In extensive simulations, NetKV outperforms traditional scheduling methods, demonstrating a mean TTFT reduction of up to 21.2% and maintaining low overhead across various conditions.
Ignoring network dynamics in LLM inference scheduling can lead to suboptimal performance, but NetKV achieves up to 21.2% faster responses by integrating network awareness into the scheduling process.
Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget. Current schedulers route on compute load and prefix-cache locality alone, ignoring the topological distance and dynamic congestion between prefill and decode instances. We close this gap with a thin operator-to-scheduler interface, the network cost oracle, and we prove that ignoring the network term renders cache-aware-only scheduling arbitrarily suboptimal as context length grows. NetKV, the O(|D|) per-request greedy that consumes this oracle, has tier rankings that are provably robust to stale telemetry. On a 64-GPU four-tier fat-tree simulator driven by Mooncake traces, NetKV reduces mean TTFT by up to 21.2% over round-robin and 17.6% over a tuned cache+load-aware scheduler, lifts SLO attainment by up to 20.1 percentage points, and keeps the Time Between Tokens overhead below 0.5 ms in every condition tested, with no changes to the transport, inference engine, or hardware.