Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper analyzes the performance of High Bandwidth Storage (HBS) and bonded global buffer memory chiplets for alleviating memory pressure during generative AI inference on resource-constrained platforms. Using a hierarchical roofline model, the authors evaluate the bandwidth and latency requirements of HBS for large language models (13B parameters) with long context lengths. They also assess the benefits of a bonded global buffer memory chiplet for smaller models (1B parameters) and provide guidance on its optimal utilization.
High Bandwidth Storage can enable interactive throughput for large language models on resource-constrained devices, but only if bandwidth and latency requirements are carefully met.
The rise of generative AI workloads, particularly language model inference, is intensifying on/off-chip memory pressure. Multimodal inputs such as video streams or images and downstream applications like Question Answering (QA) and analysis over large documents incur long context lengths, requiring caching of massive Key and Value states of the previous tokens. Even a low degree of concurrent inference serving on resource-constrained devices, like mobiles, can further add to memory capacity pressure and runtime memory management complexity. In this paper, we evaluate the performance implications of two emerging technology solutions to alleviate the memory pressure in terms of both capacity and bandwidth using a hierarchical roofline-based analytical performance model. For large models (e.g., 13B parameters) and context lengths, we investigate the performance implications of High Bandwidth Storage (HBS) and outline bandwidth/latency requirements to achieve an acceptable throughput for interactivity. For small models (e.g., 1B parameters), we evaluate the merit of a bonded global buffer memory chiplet and propose how to best utilize it.