Search papers, labs, and topics across Lattice.
SiDP is introduced as a memory-efficient data-parallel paradigm for offline LLM inference, addressing the limitations of standard data parallelism (DP) and model parallelism (MP). It organizes model weights as a distributed pool, with each layer owned by a single GPU, accessed via Weight-as-a-Service (WaS) for large batches and Compute-as-a-Service (CaS) for small batches. Evaluations on various NVIDIA GPUs and LLMs demonstrate up to 1.8x increase in usable KV capacity and up to 1.5x higher end-to-end throughput compared to vLLM.
Squeeze 50% more throughput from your GPUs on offline LLM inference by trading off local weight storage for a distributed, on-demand weight pool.
The rapid adoption of large language models (LLMs) has shifted a substantial portion of inference workloads into throughput-oriented offline regimes, where fully utilizing GPU compute requires large batch sizes. However, existing deployments face a structural tension. Data parallelism (DP) scales throughput well but replicates model weights, leaving limited GPU memory for key-value (KV) cache and constraining batch size. Model parallelism reduces per-device weights, but requires fine-grained synchronization that erodes DP's independence and scheduling flexibility. We present SiDP, a memory-efficient data-parallel paradigm for offline LLM inference that treats weights as a bandwidth-backed shared resource inside a DP group. Instead of storing the full model on every GPU, SiDP organizes weights as a distributed pool: each layer is owned by a single GPU, and other replicas access its weights on demand via two complementary execution modes: a Weight-as-a-Service (WaS) mode that streams remote weights over NVLink into a small cache in the large-batch regime, and a Compute-as-a-Service (CaS) mode that ships activations to owners in the small-batch tail. Evaluated on NVIDIA H20, H200, and B200 GPUs with Qwen3-32B, Qwen2.5-72B, and Llama-3.1-70B, SiDP increases usable KV capacity by up to 1.8x under the same configurations, and converts this into up to 1.5x higher end-to-end throughput over baselines (vLLM) for offline workloads.
