Search papers, labs, and topics across Lattice.
This paper introduces a privacy-centric edge-cloud collaborative inference framework for large language models (LLMs) that addresses the challenges of response latency, hardware limitations, and user privacy. By leveraging endpoint-authenticated KV caching and distributing tasks between local devices and the cloud, the framework achieves significant reductions in per-token latency and downlink payloads while maintaining performance comparable to full cloud inference. Evaluations show a reduction in latency by up to 46.1% and downlink payloads by up to 67.4% compared to traditional split inference methods.
Achieving up to 46.1% lower latency and 67.4% reduced data transmission, this framework redefines efficient and privacy-aware LLM inference across diverse devices.
On-device LLM inference faces a trilemma of response latency, limited hardware resources and user privacy. Full cloud inference delivers strong computing power but exposes user prompts and dialogue data, while standalone on-device inference is unfeasible for most consumer and embedded edge devices. This paper presents a privacy-centric edge-cloud collaborative LLM inference framework built on endpoint-authenticated KV cache. Local endpoints handle input preprocessing, embedding computation, adaptive feature optimization, KV cache authentication, speculative decoding and low-dimensional model head calculation, while the cloud conducts authenticated decoder inference, KV cache management, token verification and high-dimensional vocabulary projection. Endpoints fuse partial outputs, apply language-adaptive masking and sample target tokens. All transmitted data and truncated logits are quantized and AES-GCM encrypted for privacy, with core lightweight modules, draft parameters and cache access policies kept local to avoid leakage. The framework supports heterogeneous devices including CPU-only, GPU-equipped and embedded devices via optimized streaming, batching and quantized ONNX deployment. Evaluations demonstrate that the framework reduces per-token latency by up to 46.1\% and downlink payloads by up to 67.4\% over baseline split inference, retaining comparable performance to full cloud inference.
