Search papers, labs, and topics across Lattice.
HeteroMosaic is a novel scheduling framework designed to optimize energy-efficient inference for large language models (LLMs) on heterogeneous edge system-on-chips (SoCs) by leveraging the combined capabilities of CPUs, integrated GPUs, and neural processing units. By employing a heterogeneous roofline model and decomposing inference tasks into dependency-preserving micro-batches, the framework effectively coordinates cross-accelerator execution while addressing practical challenges such as memory contention and device variation. The implementation demonstrates significant performance improvements, achieving up to 2.05X speedup and reducing energy consumption by 45.3% compared to existing frameworks and baselines.
HeteroMosaic achieves up to 2.05X faster inference and 45.3% lower energy consumption by intelligently coordinating heterogeneous resources in edge AI systems.
Modern edge system-on-chips (SoCs) combine CPUs, integrated GPUs (iGPUs), and neural processing units (NPUs), yet existing LLM runtimes typically make coarse device-level decisions or optimize operators in isolation. As a result, they underutilize heterogeneous resources, particularly on unified-memory platforms where performance depends on both device placement and task-graph coordination. We present HeteroMosaic, a heterogeneity-first scheduling framework for edge LLM inference. HeteroMosaic first uses a heterogeneous roofline model to identify when combining iGPU and NPU execution is beneficial. It then decomposes inference into dependency-preserving micro-batches that expose cross-accelerator overlap and applies trace-guided co-optimization of scheduling and device allocation under practical effects such as memory contention, DVFS, device variation, and NPU runtime overheads. We implement HeteroMosaic in PyTorch C++ and evaluate it on three AMD Ryzen AI platforms spanning NPU-heavy, balanced, and iGPU-heavy designs. On the balanced platform, HeteroMosaic achieves up to 1.73X speedup over an iGPU baseline, 1.78X over an NPU baseline, and 2.05X over frameworks such as \texttt{llama.cpp}, while reducing energy by up to 45.3%. It also improves performance over prior heterogeneous edge AI solutions by up to 2.35X.
