Search papers, labs, and topics across Lattice.
This paper introduces SPADE, a distributed inference framework that leverages speculative decoding to enhance the deployment of large language models (LLMs) across edge and cloud environments. By utilizing a compact draft model on the edge to generate candidate tokens and a larger verifier model in the cloud to validate these tokens, the framework significantly reduces cloud queries while maintaining accuracy. Experimental results show that SPADE achieves a 76% reduction in cloud model calls without any loss in performance, making it a cost-effective solution for LLM deployment.
SPADE slashes cloud model calls by 76% while preserving accuracy, revolutionizing the deployment of large language models in edge environments.
Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, \our{}, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that \our{} reduces the cloud model calls by $76\%$ with zero loss in accuracy as compared to the full model.
