Search papers, labs, and topics across Lattice.
This paper introduces NeuroPrefetcher, an innovative inference system designed for large language models (LLMs) that addresses the challenge of deploying models larger than available memory on edge devices. By leveraging the strong temporal locality of MLP activity during autoregressive decoding, NeuroPrefetcher employs predictive delta prefetching to efficiently manage weight fetching from storage, significantly improving inference speed. The results demonstrate a remarkable 7.9-12.0x speedup over existing methods like llama.cpp under constrained memory conditions, showcasing its effectiveness in real-world applications.
Predictive delta prefetching can achieve up to 12x faster LLM inference on edge devices, even when models exceed memory limits.
Deploying large language models on edge devices is increasingly limited by a widening gap between model size and available memory. Existing approaches such as quantization, smaller models, and offloading can raise the effective memory limit, but they still assume that the model can be compressed or partitioned to fit within some budget. We target the harder model-exceeds-memory setting, in which the model remains larger than resident memory throughout execution and storage becomes an active source of weights on the critical path. We observe that MLP activity during autoregressive decoding has strong temporal locality: approximately 82-85% of active neurons persist from one token to the next. This means that most sparse weights needed for the current token are already resident, and only the newly needed rows must be fetched from storage. We present NeuroPrefetcher, a storage-backed LLM inference system that exploits this property through predictive delta prefetching. After layer 0, a single GPU-resident predictor, occupying 2.86% of base model parameters, predicts sparse activity for all downstream MLP layers in one forward pass. The runtime compares these predictions against resident GPU buffers and issues application-scheduled NVMe reads only for incoming delta rows, replacing reactive operating-system demand paging with explicit, model-aware weight movement. On real unified-memory edge hardware, NeuroPrefetcher achieves 7.9-12.0x speedup over llama.cpp across constrained memory budgets.
