Search papers, labs, and topics across Lattice.
This study evaluates the performance of High-Bandwidth Flash (HBF) as a replacement for SSDs in serving key-value (KV) workloads for large language models (LLMs). Contrary to expectations, the integration of HBF leads to a significant increase in end-to-end latency (2–5.5 times) and a decrease in maximum goodput (1.1–2.7 times), primarily due to the mismatch between HBF's capabilities and the read/write patterns of transient KV workloads. The findings highlight that while HBF has potential, its effectiveness is contingent on careful system design that considers reuse-aware placement and thermal management, rather than simply substituting it for SSDs.
Replacing SSDs with High-Bandwidth Flash in LLM serving can paradoxically slow down performance by over 5 times due to mismatched workload characteristics.
A faster storage device should make serving faster. We find the opposite. High-Bandwidth Flash (HBF) stacks NAND behind a wide, package-local interface, promising flash-scale capacity with far lower read latency and higher bandwidth than an SSD. The obvious move is to keep an SSD-style Mooncake KV-offloading stack and swap in HBF underneath. We built that system and measured it: an extended TokenSim, four complete two-hour Qwen-Bailian production traces, five dense and mixture-of-experts models, and H100/B200 profiles. The upgrade backfires. Average end-to-end latency rises 2--5.5$\times$ and maximum SLO goodput falls 1.1--2.7$\times$ across H100 and B200, so the faster device yields a slower system. A cost-benefit model explains the paradox: a faster far tier pays off only when read I/O is the bottleneck, reads outweigh writes, and delivered bandwidth is sustainable. Transient KV violates all three at once. Buying flash through the package costs GPU near-tier capacity and bandwidth, while HBF's own read/write latency barely matters: scaling it 3.75$\times$ moves latency less than 1\%. Worse, the two-tier hierarchy keeps reuse in the near tier and hands HBF a relentless write-heavy stream. Writes outnumber reads on every trace, so a 3D-ICE model shows the stack hits its thermal limit well below peak bandwidth, and a TLC tier wears out sooner than the SSD pool it replaced. The device is fine; the drop-in deployment is not. HBF sucks as an SSD replacement for transient KV, but earns its place in LLM serving when used selectively with reuse-aware placement, write budgeting, and thermal coordination.
