Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a transition-aware backend dispatch method for optimizing large language model (LLM) inference on edge platforms by dynamically selecting execution backends based on operator features and previously chosen backends. By analyzing ordered traces from full-model inference of seven transformer models, the authors benchmark four operator classes across various execution environments, leading to significant reductions in latency, energy consumption, and energy-delay product. The results show that this approach outperforms static backend assignments and independent per-operator selections, enhancing efficiency in edge LLM deployments.
Transition-aware backend dispatch can cut inference latency and energy use by over 17% on edge devices, making LLMs more efficient than ever.
Efficient large language model (LLM) inference on edge platforms is limited not only by model size, but also by shape-dependent performance differences across execution backends. Static backend assignment cannot exploit this variation, while independent per-operator selection can introduce costly device and framework switches. This paper presents a transition-aware backend dispatch approach for edge transformer inference. The approach combines current operator features with the previously selected backend to preserve beneficial shape-specific choices while avoiding unnecessary transitions. Ordered traces are collected from full-model inference runs of seven transformer models, and four common operator classes are benchmarked across PyTorch eager CPU, PyTorch eager CUDA, and ONNX Runtime CPU on an NVIDIA Jetson platform. The dispatch policies are evaluated through measurement-backed trace replay using observed operator costs and transition costs measured from actual backend switches. Supported operators are selected dynamically, while operators outside the dispatch scope retain a static assignment. Across 9,584 ordered operator instances and 278 exact shape groups, transition-aware dispatch reduces replayed latency, energy, and energy-delay product relative to the best static policy by 17.4%, 14.4%, and 28.5% on average, respectively. It also reduces switching relative to operator-local selection. Leave-one-model-out evaluation improves all three objectives for six of seven held-out models and improves energy for all seven. These results demonstrate that incorporating operator shape and backend-transition context can improve selective backend dispatch for edge transformer workloads.