Search papers, labs, and topics across Lattice.
This paper introduces AceSpec, an innovative asymmetric edge-cloud collaborative framework designed to enhance communication-efficient inference of Large Language Models (LLMs) on edge devices. By leveraging a probabilistic state cache and an asymmetric communication protocol, AceSpec mitigates the issues of pipeline stalls and bandwidth limitations commonly faced in edge-cloud setups. The results indicate that AceSpec can achieve up to a 3.52脳 throughput speedup while maintaining robust performance under constrained WAN conditions, significantly improving the efficiency of LLM deployment on edge devices.
AceSpec transforms edge-cloud LLM inference by turning catastrophic network stalls into efficient local memory lookups, achieving a remarkable 3.52脳 speedup even in low-bandwidth environments.
Deploying Large Language Models (LLMs) on edge devices typically relies on model compression or split inference. However, compression degrades reasoning capabilities, while split inference suffers from severe Wide Area Network (WAN) communication bottlenecks. Edge-cloud speculative decoding emerges as a promising alternative, leveraging an edge small model to draft tokens for cloud verification. Yet, over volatile WANs, inevitable prediction rejections trigger catastrophic pipeline stalls and network-wide rollbacks, neutralizing collaborative gains. To overcome this, we propose AceSpec, an asymmetric edge-cloud collaborative framework. AceSpec utilizes un-saturated edge compute to proactively construct a probabilistic state cache, effectively transforming network-wide pipeline flushes into $\mathcal{O}(1)$ local memory lookups. To preserve bandwidth, it employs an asymmetric communication protocol that transmits minimal main-chain indices uplink and compact sparse distributions downlink. Furthermore, we introduce a network-aware, Lagrangian-optimized resource allocation strategy that dynamically maximizes the local cache hit rate. Evaluations demonstrate that AceSpec achieves up to a 3.52$\times$ throughput speedup and exhibits exceptional bandwidth immunity, sustaining near-peak inference performance even under severely constrained 50 Kbps WAN conditions.
