Search papers, labs, and topics across Lattice.
This paper introduces RAC, a reference-aware codec designed to enhance communication efficiency in split inference for large language models (LLMs) by minimizing the transfer of boundary hidden states. By leveraging historical token retrieval and lightweight causal predictors, RAC significantly reduces the time to first token (TTFT) and time per output token (TPOT) while maintaining quality across various model-link pairs. The results show that RAC achieves a mean TTFT and TPOT improvement of up to 2.72x and 2.79x, respectively, with minimal impact on task performance scores.
Achieving up to 2.72x faster inference times, RAC transforms split LLM deployment by slashing communication bottlenecks without sacrificing performance.
Large language model (LLM) agents repeatedly process long, privacy-sensitive contexts, while cloud-only deployment exposes user data beyond the trusted endpoint and fully local deployment often requires costly hardware. Split inference offers a middle ground by executing the model head, tail, and tools locally and the middle layers in the cloud, but its local-cloud-local path transfers boundary hidden states at every invocation and creates a critical communication bottleneck. We present \system, a reference-aware codec that retrieves exact-token historical spans for prefill uplinks, reuses the reconstructed uplink state for same-round prefill downlinks, and generates boundary-specific decode references with lightweight causal predictors. RAC applies grouped affine alignment and calibrated residual quantization with optional prefill outliers, while sender-side wire-format reconstruction synchronizes subsequent references and offline calibration accounts for quality and packed representation costs. Across three models and nine evaluated model-link pairs, Raw-to-RAC mean time to first token (TTFT) and time per output token (TPOT) ratios are 1.24-2.72$\times$ and 1.01-2.79$\times$, while the 12 non-perplexity task-score changes range from $-0.40$ to $+2.50$ points.
