Search papers, labs, and topics across Lattice.
This paper introduces a novel sequential-dispatch measurement method to accurately assess the per-operation dispatch overhead in WebGPU for Large Language Model (LLM) inference. The authors reveal that naive measurement techniques overestimate dispatch costs by conflating them with synchronization, leading to a mischaracterization of performance bottlenecks. Their findings indicate that at batch size 1, reducing dispatch count is crucial for optimizing LLM inference in WebGPU, highlighting the need for dispatch amortization strategies in both inference engines and the WebGPU specification.
Reducing dispatch count is the key to unlocking efficient LLM inference in WebGPU, not improving kernel quality.
Large Language Models are deployed to multiple types of environments, from internet browsers to edge devices, and WebGPU serves as a modern cross-platform standard. The engines for browser-based LLM inference have proliferated, yet the overhead of WebGPU per-operation dispatch remains poorly characterized. In this work, we introduce a sequential-dispatch measurement method and show that naive single-operation measurements overestimate per-dispatch cost by conflating dispatch with synchronization. Using our method, we measure the per-dispatch cost and show that it is independent of data type used. We show that the dispatch overhead, not kernel quality, is the bottleneck at batch size 1, and isolate the dispatch count as the cause. Therefore, we conclude that at batch size 1, the effective approach to LLM inference optimization in WebGPU is reducing dispatch count. Our findings point to dispatch amortization, in the inference engines and in the WebGPU specification, as a path to practical browser-based inference.
