Search papers, labs, and topics across Lattice.
FASER introduces a fine-grained speculative decoding (SD) management system for dynamic LLM serving that addresses the limitations of coarse-grained approaches. It dynamically adjusts speculative token length per request and prunes rejected tokens early, minimizing computational waste. By breaking the verification phase into frontiers and overlapping them with the draft phase via spatial multiplexing, FASER achieves better resource utilization. FASER improves throughput by up to 53% and reduces latency by up to 1.92x compared to existing SD systems in vLLM.
Fine-grained management of speculative decoding phases can boost LLM serving throughput by over 50% and cut latency nearly in half.
Speculative decoding (SD) is a widely used approach for accelerating decode-heavy LLM inference workloads. While online inference workloads are highly dynamic, existing SD systems are rigid and take a coarse-grained approach to SD management. They typically set the speculative token length for an entire batch and serialize the execution of the draft and verification phases. Consequently, these systems fall short at adapting to volatile online inference traffic. Under low load, they exhibit prolonged latency because the draft phase blocks the verification phase for the entire batch, leaving GPU computing resources underutilized. Conversely, under high load, they waste computation on rejected tokens during the verification phase, overloading GPU resources. We introduce FASER, a novel system that features fine-grained SD phase management. First, FASER minimizes computational waste by dynamically adjusting the speculative length for each request within a continuous batch and by performing early pruning of rejected tokens inside the verification phase. Second, FASER breaks the verification phase into frontiers, or chunks, to overlap them with the draft phase. This overlap is achieved via fine-grained spatial multiplexing with minimal resource interference. Our FASER prototype in vLLM improves throughput by up to 53% and reduces latency by up to 1.92$\times$ compared to state-of-the-art systems.
