Search papers, labs, and topics across Lattice.
AsymSpec introduces an innovative approach to cloud-edge speculative decoding that optimizes communication and processing efficiency over asymmetric networks. By implementing a novel asymmetric verification protocol and a confirmed-prefix pipeline, it minimizes idle time and maximizes edge compute utilization, addressing the challenges of uplink-gated verification and invalidated dependent work. The end-to-end evaluation demonstrates that AsymSpec achieves an impressive 2.82-28.03 times higher output-token throughput compared to existing baselines across various network conditions and workloads.
AsymSpec boosts output-token throughput by up to 28 times by cleverly optimizing communication between edge and cloud models.
Cloud-edge speculative decoding places a lightweight draft model at an edge gateway and a higher-quality target model in the cloud, but inserts communication into every speculative block. Under a constrained uplink, candidate messages may queue while the verifier is idle. Stop-and-wait scheduling leaves edge compute underutilized; optimistic same-request runahead can waste work when a rejection or an unexpected bonus token invalidates dependent drafts. We present AsymSpec, which addresses uplink-gated verification and invalid dependent work with two corresponding mechanisms. Its asymmetric verification protocol keeps the common-path acceptance upload compact and moves richer, rejection-only correction information to the downlink. A total-variation (TV) certificate for the residual distribution determines whether a small target top-K response suffices; if not, the protocol progressively escalates through proposal-based exact recovery before falling back to the full distribution. Its confirmed-prefix pipeline exposes only independent, valid requests to the edge scheduler and lets the cloud re-batch arrived blocks, hiding verification waits when another confirmed-prefix request is ready without using same-request runahead. Across three draft-target pairs, two workloads, and three asymmetric network profiles, our end-to-end evaluation shows that AsymSpec delivers 2.82-28.03$\times$ the output-token throughput of the strongest baseline.
