Search papers, labs, and topics across Lattice.
To resolve the tradeoff between kernel launch overhead and inter-token latency spikes in LLM serving, SLOWeave dynamically sizes prefill chunks to fill available time headroom before the earliest active decode deadline. The scheduler relies on a monotonic iteration-cost model evaluated via logarithmic-time search, provably maximizing prefill progress while strictly preserving per-request decode latency targets without manual tuning. Under a strict 10ms time-per-output-token SLO on GPU runtimes, this deadline-aware adaptation yields up to a 3.3脳 goodput improvement over the strongest fixed-chunk baselines.
Fixed-size chunked prefill forces an unnecessary compromise between decode latency and launch overhead: dynamically sizing chunks to fit active decode deadlines boosts serving goodput by up to 3.3脳 under tight latency SLOs.
Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3$\times$ and 2.4$\times$, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.
