Search papers, labs, and topics across Lattice.
This paper introduces PicoSpec, a pipelined collaborative speculative decoding framework for efficient edge-cloud LLM inference. PicoSpec uses an asynchronous pipeline to concurrently execute a Small Language Model (SLM) on the edge and a Large Language Model (LLM) in the cloud, overcoming the mutual waiting problem in vanilla speculative decoding. By employing separate rejection sampling with sparse compression, PicoSpec reduces communication latency by transmitting compressed vocabulary distributions, achieving up to 2.9x speedup compared to baselines.
Achieve nearly 3x faster LLM inference by intelligently splitting the workload between edge devices and the cloud, without any training.
Recent advancements and widespread adoption of Large Language Models (LLMs) in both industry and academia have catalyzed significant demand for LLM serving. However, traditional cloud services incur high costs, while on-device inference alone faces challenges due to limited resources. Edge-cloud collaboration emerges as a key research direction to combine the strengths of both paradigms, yet efficiently utilizing limited network bandwidth while fully leveraging and balancing the computational capabilities of edge devices and the cloud remains an open problem. To address these challenges, we propose Pipelined Collaborative Speculative Decoding Framework (PicoSpec), a novel, general-purpose, and training-free speculative decoding framework for LLM edge-cloud collaborative inference. We design an asynchronous pipeline that resolves the mutual waiting problem inherent in vanilla speculative decoding within edge collaboration scenarios, which concurrently executes a Small Language Model (SLM) on the edge device and a LLM in the cloud. Meanwhile, to mitigate the significant communication latency caused by transmitting vocabulary distributions, we introduce separate rejection sampling with sparse compression, which completes the rejection sampling with only a one-time cost of transmitting the compressed vocabulary. Experimental results demonstrate that our solution outperforms baseline and existing methods, achieving up to 2.9 speedup.
