Search papers, labs, and topics across Lattice.
This paper introduces NELSSA, a heterogeneous LLM serving system that combines GPUs with Processing-near-Memory (PNM) accelerators to address the inefficiencies of mixed-length workloads in LLM applications. By implementing length-based request placement, NELSSA routes short-context requests to GPUs and long-context requests to PNM, while also allowing for runtime migration to manage dynamic context growth. The results show that NELSSA achieves up to 5.5x improvement in decode throughput and reduces P99 latency by up to 15x compared to traditional GPU-only systems, highlighting its potential for scalable LLM infrastructures.
NELSSA achieves a staggering 5.5x increase in decode throughput for mixed-length LLM workloads by intelligently integrating GPUs with PNM accelerators.
Modern LLMs and their agentic applications are broadening the range of serving workloads, spanning context lengths from a few hundred tokens to hundreds of thousands. As these requests frequently interleave within the same serving window, LLM serving systems must handle highly heterogeneous mixed-length workloads. Such mixed-length workloads expose fundamental inefficiencies in GPU-centric serving architectures, whose throughput depends on large, memory-constrained batches. In this paper, we present NELSSA, an LLM serving system that integrates GPUs with real-world Processing-near-Memory (PNM) accelerator devices to efficiently support mixed-length workloads. NELSSA employs length-based request placement to route short-context requests to GPUs and long-context requests to the PNM tier, incorporating runtime migration to accommodate dynamic context growth without recomputation. We prototype NELSSA as an end-to-end system, implementing device-level sparse attention on PNM, GPU decode kernels, and a host-side runtime that orchestrates scheduling and cross-tier memory movement over a CXL-enabled infrastructure with RPC and RDMA support. Across mixed-length LLM workloads, NELSSA improves decode throughput by up to 5.5x in tokens/sec and reduces P99 latency by up to 15x compared to GPU-only baselines. Our end-to-end prototype and experimental results suggest that integrated GPU-PNM serving, enabled by CXL-based disaggregation, is a promising system paradigm for scalable and flexible LLM infrastructures that support evolving workloads.
