Search papers, labs, and topics across Lattice.
This paper introduces a hybrid theoretical and empirical approach for determining the optimal allocation of prefill and decode hardware resources in disaggregated LLM inference, considering throughput requirements, SLOs, and request characteristics. The method uses M/M/1 queuing theory to model prefill throughput based on TTFT benchmarks and derives decode throughput from empirical measurements of decode batch sizes that meet TPOT requirements. Experiments validate the method's accuracy in predicting optimal resource allocation for real-world LLM inference.
Stop guessing at prefill/decode resource allocation: this method accurately predicts the optimal split for disaggregated LLM inference, balancing throughput and SLOs.
Prefill-Decode (P/D) disaggregation has emerged as a widely adopted optimization strategy for Large Language Model (LLM) inference. However, there currently exists no well-established methodology for determining the optimal number of P/D hardware resources, subject to constraints on total throughput, service level objectives (SLOs), and request characteristics - specifically input and output lengths. To address this gap, we propose a hybrid approach that combines theoretical modeling with empirical benchmarking. First, we present a theoretical model for calculating P/D resource counts, which is based on total throughput requirements, request input and output lengths, as well as prefill and decode throughput. Then, to obtain the actual prefill and decode throughput under SLO constraints, we model the prefill process using M/M/1 queuing theory, deriving the achieved prefill throughput from the benchmarked maximum prefill throughput and Time-To-First-Token (TTFT). For the decode phase, we determine the decode batch sizes that meet Time-Per-Output-Token (TPOT) requirements and obtain the corresponding decode throughput through empirical measurements. Our experimental results demonstrate that the proposed method can accurately predict optimal P/D resource allocation in real-world LLM inference scenarios.
