Search papers, labs, and topics across Lattice.
FleetSieve optimizes tensor-parallel degrees and replica counts for LLM serving fleets by selectively profiling configurations based on their expected impact on resource allocation and service level objectives (SLOs). By modeling capacity and tail latency jointly, FleetSieve efficiently narrows down decision gaps, achieving significant performance improvements over traditional random profiling methods. In experiments, FleetSieve demonstrated a 5.4% mean saving in GPU resource usage compared to random profiling, while ensuring compliance with strict latency requirements.
FleetSieve cuts GPU resource usage by over 5% while ensuring LLM latency meets stringent service level objectives.
Choosing tensor-parallel (TP) degrees and replica counts for an LLM serving fleet is difficult because performance is not monotonic in TP and the feasible choice can change with load. Exhaustive profiling resolves this uncertainty, but measures many configurations that do not affect the final resource allocation. We present FleetSieve, which selects measurements according to their expected effect on a resource-coupled, SLO-aware fleet decision. FleetSieve models capacity and tail latency jointly, compares conservative and optimistic allocations, and stops when their remaining decision gap is below a specified tolerance. On a fixed H100 measurement grid for a 31B-parameter open-weight model, FleetSieve reaches the oracle aggregate decision using 22,200 GPU-seconds, 6.9% less than uniform random profiling in the fixed comparison. Across 200 random reveal orders, its mean saving over random profiling is 5.4% (95% bootstrap CI: 3.5-7.2%). The fixed-comparison saving is 21.5% for Chat, while FleetSieve does not use the fewest GPU-seconds for Code. Joint capacity and tail modeling also avoids selecting a configuration whose 46.4-second completion p99 violates a 30-second SLO. In a 16-GPU allocation, an incorrect sparse-profile decision loses up to 1.93 requests/s and 12.4 percentage points of max-min fulfillment. Boundary repeats and BurstGPT measurements support the observed load-dependent tail-latency mechanism.
