Search papers, labs, and topics across Lattice.
This paper addresses the challenge of maintaining inference continuity in edge LLMs during user handovers by introducing an importance-aware key-value (KV) cache transfer method. By prioritizing the transmission of the most informative cache entries, the authors transform the transfer process into a multi-user backhaul allocation problem that maximizes average accuracy while minimizing bandwidth saturation. The proposed allocator achieves over 93.7% average accuracy in a 500ms transfer window, closely approaching the theoretical upper bound of performance.
Prioritizing KV cache entries by importance can achieve over 93.7% accuracy during edge LLM handovers, significantly optimizing bandwidth usage.
Edge LLMs must preserve inference continuity when a user hands over between edge nodes, requiring key-value (KV) cache transfer to the target node. However, simultaneous handovers saturate the backhaul, preventing full cache delivery within the mobility-imposed transfer window. Rather than allocating bandwidth as if all cache entries were equally valuable, we order each user's KV cache by importance and transmit only its most informative fraction, turning token-level sparsity into communication savings. We cast the transfer as a multi-user backhaul allocation problem that maximizes average accuracy across users. Each user's partial-cache accuracy serves as its utility: a sigmoid that fits measurements on the RULER benchmark with $R^2>0.99$ across models and context lengths. Because importance ordering front-loads the high-value entries, the concave region of the accuracy curve spans nearly the entire cache. Our proposed allocator keeps served users within this region, making each per-slot allocation problem convex. The optimum is derived via a closed-form weighted water-filling solution that generalizes information-theoretic water-filling and enables online scheduling. The proposed allocator attains over 93.7% average accuracy in a 500ms transfer window, within 0.5pp of the full-cache ceiling, and reaches 98.2-99.5% of a clairvoyant upper bound.
