Search papers, labs, and topics across Lattice.
This paper introduces ITME (Inference Tiered Memory Expansion), a novel architecture that utilizes CXL-hybrid memory to enable TB-scale byte-addressable remote memory expansion for Large Language Models (LLMs). By optimizing data movement based on deterministic access patterns of model weights and prefix caches, ITME simplifies the software stack and enhances cost-efficiency in shared context infrastructures. Experimental validation with production-grade hardware shows that ITME can achieve up to a 35.7% improvement in throughput, addressing the growing demands of long-context workloads in AI applications.
ITME achieves a remarkable 35.7% throughput improvement by leveraging CXL-hybrid memory to expand remote memory for LLMs, tackling the challenges of disaggregated shared storage.
The rapid shift toward agentic and long-context workloads in Large Language Models (LLMs) is pushing the industry beyond the capacity of individual servers toward disaggregated shared storage to handle TB-scale context states. This movement has led to the emergence of specialized shared context layers designed to externalize and share cumulative inference states across distributed clusters. While offloading to a data processing unit (DPU) within just-a-bunch-of-flash (JBOF) architectures accelerates NVMe-over-fabrics (NVMe-oF) target processing, the need for sophisticated software-level optimization and cost-efficiency burdens remain significant. Consequently, the ideal architecture for scaling this shared context infrastructure is still an active area of exploration. In this paper, we propose ITME (Inference Tiered Memory Expansion), which leverages a CXL-hybrid memory to present a massive, TB-scale byte-addressable remote memory expansion. This approach enables cost-efficient scaling and simplifies the software stack through direct byte-addressability, effectively addressing the challenges of shared context infrastructure. Our key insight is that the deterministic access patterns of voluminous model weights and prefix caches enable the system to proactively manage data movement across the memory-storage hierarchy. We validate ITME by evaluating its performance potential with production-grade SK Hynix CMM and PCIe Gen5 NVMe SSDs, while further demonstrating its functional feasibility through an FPGA-based hardware prototype. Overall, ITME enhances conventional CPU-offloading by providing additional remote memory expansion to accommodate large KV cache footprints beyond host memory limits, achieving up to a 35.7\% throughput improvement.
