Search papers, labs, and topics across Lattice.
This paper introduces OUTLETS, a novel method for predicting output lengths in Large Language Models (LLMs) by leveraging latent representations from speculative decoding backbones. By repurposing the draft decoder's outputs, OUTLETS achieves a significant reduction in mean absolute error (MAE) compared to existing length prediction methods, while maintaining low latency. The implementation of OUTLETS in saturated disaggregated serving environments results in a 34.8% decrease in short-request P99 latency, enhancing resource allocation and scheduling efficiency.
Repurposing speculative decoding backbones for output-length prediction slashes short-request latency by nearly 35%.
The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce OUTLETS (Output-Length Prediction from Speculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by 34.8%.
