Search papers, labs, and topics across Lattice.
This paper introduces Tropical, a novel SLO-aware multiplexing strategy designed to optimize latency in disaggregated large language model (LLM) serving by balancing queuing times and interference between the prefill and decode phases. Through a comprehensive analysis, the authors identify that while disaggregated serving minimizes interference, it suffers from high queuing times, whereas non-disaggregated serving reduces queuing but increases interference. Tropical effectively combines the strengths of both approaches, achieving up to 2.09 times more requests within a 90% SLO attainment and significantly improving P90 TTFT and TPOT metrics compared to existing systems.
Tropical achieves a remarkable balance in LLM serving, outperforming both disaggregated and non-disaggregated systems by enhancing request throughput while maintaining low latency.
To guarantee service quality in transformer based large language model (LLM) serving, it is essential to meet the latency constraints of both the prefill phase (measured by Time-to-First-Token, TTFT) and the decode phase (measured by Time-per-Output-Token, TPOT). Non-disaggregated serving places prefill and decode on the same worker, while disaggregated serving places the prefill and decode on isolated workers. However, no single architecture excels in both TTFT and TPOT metrics. After conducting a root cause analysis, we concluded that indisaggregated LLM serving, prefill execution has minimal interference with decode execution but result in high queuing times. In contrast,non-disaggregated LLM serving effectively reduces queuing times but introduces significant interference between prefills and decodes. In order to leverage the best aspects of both non-disaggregated anddisaggregated LLM serving, we have designed and implemented Tropical.Tropical introduces an sevice-level objectives (SLO)-aware multiplexing strategy that balances the queuing time and the interference, enabling the LLM serving to achieve high TTFT and TPOT SLOs simultaneously. Our evaluation of real-world datasets reveals that Tropical outperforms both state-of-the-art non-disaggregated and disaggregated LLM serving systems, achieving up to 2.09 more requests within a 90% SLO attainment. Specially, compared to the disaggregated LLM serving system, Tropicalimproves P90 TTFT performance by 9 with only an 15% reduction in P90 TPOT. Against the non-disaggregated LLM serving systems, Tropicaldelivers a 2.8 performance improvement in P90 TPOT while maintaining the same P90 TTFT.
