Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DOPS (dynamic operator scheduling), a closed-loop framework that optimizes operator scheduling and weight layouts for Large Language Model (LLM) inference across heterogeneous systems. By employing a stage-aware directed acyclic graph (DAG) and integrating the Bifocal scheduler with the Weight Layout Arbiter (WLA), the authors achieve significant speedups in LLM inference, ranging from 1.20× to 2.23× compared to traditional prefill-decode methods. The results highlight the importance of hardware-aware scheduling and layout strategies in improving the efficiency of LLM deployment on diverse hardware platforms.
Achieving up to 2.23× speedups in LLM inference by optimizing operator scheduling and weight layouts could revolutionize how we deploy models on heterogeneous hardware.
Prefill-decode disaggregation (PD) and roofline-based operator placement are common strategies for partitioning Large Language Model (LLM) inference across heterogeneous systems, but they are often insufficient in practice. End-to-end latency also depends on workload shape, runtime device contention, and persistent weight layout. We present DOPS (dynamic operator scheduling), a hardware-aware, closed-loop framework that jointly optimizes operator scheduling and blockwise weight layouts. DOPS constructs a stage-aware directed acyclic graph (DAG) and integrates two components: the Bifocal scheduler for dynamic operator-to-device placement and the Weight Layout Arbiter (WLA) for selecting hardware-efficient weight layouts under strict memory constraints. Across representative heterogeneous systems combining neural processing units (NPUs) and processing-in-memory (PIM) devices, Bifocal achieves geometric-mean speedups of 1.20$\times$ to 2.23$\times$ over the PD baseline. WLA provides an additional geometric-mean speedup of 1.28$\times$ to 1.33$\times$ over Bifocal/Linear. DOPS also supports systematic analysis of workload sensitivity and hardware scalability for LLM serving. The source code is available at https://github.com/YIAI-02/TriForm, and the visualization tool is demonstrated at https://youtu.be/Ya_oMCyYno0.