Search papers, labs, and topics across Lattice.
This paper introduces LLMVisor, a real-time latency attribution model designed for multi-tenant GPU clusters serving large language models (LLMs). By utilizing a roofline-guided approach, LLMVisor effectively decomposes batch latency into additive per-request shares, achieving high accuracy in latency attribution while operating within the scheduling loop. Evaluations demonstrate that LLMVisor significantly reduces relative error in latency attribution by up to 4.4x compared to traditional token-count baselines, even under varying workloads and batching conditions.
LLMVisor achieves up to 4.4x improvement in latency attribution accuracy for multi-tenant LLMs, revealing hidden inefficiencies in GPU resource usage.
As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control. Enabling fractional sharing of the inference engine requires a real-time, per-request attribution primitive that is accurate and light enough to run inside the scheduling loop. We present LLMVisor, a roofline-guided latency attribution model that captures the memory-bound and compute-bound phases via a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic. LLMVisor decomposes batch latency into additive, per-request shares and runs efficiently at microsecond scale. We evaluate LLMVisor across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs under varying tensor parallelism and workload mixes. Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x and 3.3x at p90 and p99, respectively, for prefill, and by up to 3.5x and 4.4x for decode, despite batching variability and sequence divergence.
