Search papers, labs, and topics across Lattice.
This paper introduces LENS, a novel latency estimator designed for predicting inference latency of Large Language Models (LLMs) on Neural Processing Units (NPUs) without requiring detailed microarchitecture or compiler information. By addressing challenges such as undisclosed microarchitecture, unpredictable compiler optimizations, and non-linear latency due to bucketing, LENS utilizes end-to-end measurements to accurately predict latency across various input-output lengths. The method was validated against NPUs from multiple vendors and achieved a mean prediction error of just 2.15%, outperforming related baseline methods.
LENS achieves an impressive 2.15% mean prediction error in latency estimation for LLM inference on NPUs, even without access to microarchitecture details.
Deploying Large Language Models (LLMs) requires exploring a large configuration space spanning parallelization strategies, batching techniques, and scheduling policies. Exhaustive measurement across this space is impractical, making latency prediction essential for system optimization. While NPUs have emerged as accelerators designed for LLM inference, no prediction methodology has been established for them. Specifically, applying prior work to LLM inference latency prediction on NPUs faces three challenges: undisclosed microarchitecture of commercial NPUs, unpredictable compiler optimizations, and latency non-linearity induced by bucketing. We present LENS, a latency estimator that predicts NPU inference latency without information on the microarchitecture or compiler, and captures the non-linear latency induced by bucketing. LENS profiles each bucket with two end-to-end (E2E) measurements and composes the results to predict latency for arbitrary input-output length combinations. We validate LENS across NPUs from multiple vendors, several LLMs, and diverse workloads, achieving a mean prediction error of 2.15\%. We further compare LENS against two methodologically related baselines, confirming the validity of its approach.
