Search papers, labs, and topics across Lattice.
This paper introduces HMA-Serve, an innovative disaggregated serving system that optimally utilizes memory-heterogeneous accelerators by pairing GDDR-based accelerators for the compute-bound prefill phase with HBM-based GPUs for the memory-bound decode phase. By implementing phase-wise quantization, a compute-transfer pipeline, and deferred dequantization, HMA-Serve achieves significant improvements in goodput and cost-efficiency without compromising generation quality. The results demonstrate up to 3.2 times higher goodput and 4.8 times higher goodput-per-dollar compared to existing memory-homogeneous methods across multiple model sizes and production traces.
HMA-Serve achieves 3.2x higher goodput and 4.8x better cost-efficiency by leveraging memory-heterogeneous accelerators, challenging the status quo of single-vendor LLM serving.
LLM inference comprises a compute-bound prefill phase and a memory-bound decode phase, and recent systems disaggregate them onto separate hardware. Yet today's datacenter GPUs rely on costly HBM whose bandwidth sits almost entirely idle during prefill. LLM serving across memory-heterogeneous accelerators (MemHA) pairs GDDR-based accelerators for prefill with HBM-based GPUs for decode, promising lower cost without sacrificing performance. Pushed to its most economical form, MemHA serving is inherently cross-vendor, since the best-suited chip for each phase may come from a different vendor. This breaks two assumptions that single-vendor disaggregation takes for granted -- a KV format both ends consume natively, and a shared software stack. We present \textbf{HMA-Serve}, a MemHA-centric disaggregated serving system pairing GDDR-based accelerators for prefill with HBM-based GPUs for decode efficiently. HMA-Serve achieves this through (1) phase-wise quantization, applying vendor-native low precision for high-throughput prefill while keeping decode in high-precision BF16, (2) a compute-transfer pipeline that overlaps each layer's KV cache transfer with later-layer prefill to reduce time-to-first-token (TTFT), and (3) deferred dequantization, shipping raw quantized bytes and reconstructing them lazily on the decode GPU to reduce network bandwidth and HBM usage. Across four Qwen3 models (4B--32B) and three production traces, HMA-Serve delivers up to $3.2\times$ higher goodput than state-of-the-art memory-homogeneous methods and $4.8\times$ higher goodput-per-dollar, with no measurable loss on generation-quality benchmarks.
