Search papers, labs, and topics across Lattice.
This paper introduces Talaria, a session-aware serverless serving system designed to optimize the performance of hundred-billion-parameter LLMs by addressing the inefficiencies of independent request scheduling. By integrating session continuity into the placement and admission decisions, Talaria significantly reduces session completion times, achieving speedups of 5.3x for median and 2.6x for 95th percentile completion times compared to traditional round scheduling. The system's innovative routing and session-prefill mechanisms enhance model residency and KV locality, making it particularly effective for large-scale deployments.
Talaria slashes session completion times for hundred-billion-parameter LLMs by over 5x, revolutionizing serverless model serving efficiency.
Serverless multi-model LLM systems multiplex popularity-skewed model catalogs over shared GPU pools, yet typically schedule each request independently. Tool-using agents break this abstraction: a session repeatedly calls an LLM across short tool gaps, carries a long reusable KV prefix, and is judged by session completion time (SCT). Load-only routing can separate a continuation from both its model and KV state, while round-based model multiplexing can delay even a correctly placed continuation until the target model's next slot. Both failures are especially costly for hundred-billion-parameter models: their weights constrain residency, while long-context KV is expensive to reconstruct or move. We present Talaria, a session-aware serverless multi-model serving system that makes session continuity a joint placement-and-admission decision. Its router ranks placements by model residency, KV locality, and instance pressure, while soft reservations account for likely returns in the last serving instance's admission budget. Session-prefill (SP) admits budget-eligible continuations before the active model slot closes. An instance-local substrate keeps HBM addresses stable, preserves host-restorable KV, and stages weights across model switches. On a single TP=8 server, we replay 30 SWE-Bench model-sessions (960 calls) over three models, each with more than 100B total parameters. Against an otherwise identical round scheduler with SP, host-KV restoration, and D2D staging disabled, Talaria cuts p50 SCT from 1000 s to 189 s and p95 from 2296 s to 867 s, speedups of 5.3x and 2.6x.
