Search papers, labs, and topics across Lattice.
This paper introduces DOPS (dynamic operator scheduling), a closed-loop framework that optimizes operator scheduling and weight layouts for Large Language Model (LLM) inference across heterogeneous systems. By employing a stage-aware directed acyclic graph (DAG) and integrating the Bifocal scheduler with the Weight Layout Arbiter (WLA), the authors achieve significant speedups in LLM inference, ranging from 1.20脳 to 2.23脳 compared to traditional prefill-decode methods. The results highlight the importance of hardware-aware scheduling and layout strategies in improving the efficiency of LLM deployment on diverse hardware platforms.
Achieving up to 2.23脳 speedups in LLM inference by optimizing operator scheduling and weight layouts could revolutionize how we deploy models on heterogeneous hardware.
Prefill-decode disaggregation (PD) and roofline-based operator placement are common strategies for partitioning Large Language Model (LLM) inference across heterogeneous systems, but they are often insufficient in practice. End-to-end latency also depends on workload shape, runtime device contention, and persistent weight layout. We present DOPS (dynamic operator scheduling), a hardware-aware, closed-loop framework that jointly optimizes operator scheduling and blockwise weight layouts. DOPS constructs a stage-aware directed acyclic graph (DAG) and integrates two components: the Bifocal scheduler for dynamic operator-to-device placement and the Weight Layout Arbiter (WLA) for selecting hardware-efficient weight layouts under strict memory constraints. Across representative heterogeneous systems combining neural processing units (NPUs) and processing-in-memory (PIM) devices, Bifocal achieves geometric-mean speedups of 1.20$\times$ to 2.23$\times$ over the PD baseline. WLA provides an additional geometric-mean speedup of 1.28$\times$ to 1.33$\times$ over Bifocal/Linear. DOPS also supports systematic analysis of workload sensitivity and hardware scalability for LLM serving. The source code is available at https://github.com/YIAI-02/TriForm, and the visualization tool is demonstrated at https://youtu.be/Ya_oMCyYno0.
