Search papers, labs, and topics across Lattice.
This paper explores the integration of High-Bandwidth Flash (HBF) into GPU memory hierarchies to address the growing memory constraints in LLM serving systems. By modeling the system-level benefits of increased capacity, the authors demonstrate that HBF can enhance the efficiency of mixture-of-experts (MoE) serving and multi-model scenarios by facilitating more expert replicas and minimizing model loading times. Simulation results indicate that the effectiveness of HBF hinges on maintaining the HBM-resident execution path while expanding the resident set of model weights.
HBF can significantly boost LLM serving efficiency by enabling more expert replicas and reducing loading times, all while preserving critical execution paths.
LLM serving is increasingly constrained by memory capacity as model weights, KV caches, and the number of served model variants continue to grow. This report examines High-Bandwidth Flash (HBF) as a capacity-oriented extension to HBM-based serving systems. We first discuss how HBF can be integrated into the GPU memory hierarchy without undermining the bandwidth expected by the compute die. We then model the system-level value of added capacity as expanded residency for read-mostly model-state objects. Under this view, HBF can improve MoE serving by enabling more expert replicas and can improve multi-model serving by reducing model loading and supporting hot-model replication. Our simulation results show that these benefits depend on preserving the HBM-resident execution path while using HBF to expand the resident set of model weights.
