Search papers, labs, and topics across Lattice.
This paper evaluates the performance of distributed speculative decoding (DSD) for large language models (LLMs) when the draft model is placed on an edge device while the target model remains in the cloud. The authors derive closed-form inequalities demonstrating that DSD's latency benefits are constrained by wide-area network (WAN) communication, revealing that co-located speculative decoding (SD) consistently outperforms DSD in terms of latency and communication efficiency. Ultimately, the findings suggest that DSD's advantages are primarily relevant in multi-tenant scenarios rather than single-request latency, emphasizing the need for a shift in evaluation criteria for DSD's effectiveness.
Co-located speculative decoding consistently beats distributed variants in latency, challenging the assumption that edge-cloud setups always yield faster inference.
Speculative decoding (SD) accelerates LLM inference by $1.5$-$3$ times when the draft and target models are co-located. This has motivated a distributed variant (DSD) that places the draft model on an edge device while the target stays in the cloud. We show with closed-form inequalities that DSD's per-request latency benefit is limited under WAN edge-cloud communication. If the server can host both models, co-located SD has lower latency and communication than synchronous DSD, with the same per-output FLOPs and model-weight memory. Pipelining can make DSD competitive with co-located SD only in low-RTT regimes where the round trip is shorter than the edge drafting time window; at WAN RTTs, the cloud round trip remains too large for pipelined DSD to beat co-located SD. Against cloud autoregressive decoding, DSD can reduce latency only inside a bounded window given the target-model speed, acceptance rate, and RTT. DSD is also infeasible against closed-source APIs without a verifier-only interface. The main case for DSD appears in multi-tenant capacity. Under cross-client overlap, offloading draft compute lets a saturated cloud server sustain $(1 + 纬\,t_d/t_v)$ times more concurrent clients at the same per-client rate, where $纬$ is the speculation length and $t_d, t_v$ are the per-step draft and verification times. DSD should therefore be evaluated primarily by multi-tenant capacity and server throughput, not only by single-request latency.
