Search papers, labs, and topics across Lattice.
This paper introduces a MeanField surrogate model that efficiently predicts the performance of concurrent heterogeneous AI models on shared GPUs by leveraging local configurations and aggregate GPU states, rather than modeling all interactions. This approach significantly reduces the profiling requirements, allowing for scalable runtime scheduling even as the number of co-running models increases. Experimental results demonstrate high predictive accuracy and the ability to handle complex scheduling scenarios with minimal performance loss compared to exhaustive methods.
Achieving 78,732 feasible configurations with a scheduling accuracy within 0.10% of exhaustive search, this method revolutionizes GPU resource allocation for concurrent AI workloads.
Deploying heterogeneous AI models concurrently on a shared GPU introduces resource contention that complicates runtime scheduling. While surrogate models avoid costly online benchmarking, their profiling requirements typically grow combinatorially with the number of co-running models, limiting scalability. We propose a MeanField surrogate that predicts per-model performance from local configuration and aggregate GPU state rather than explicitly modeling all joint interactions. Experiments on concurrent LLM and vision workloads across $N \in \{2,3,4,5,6\}$ show high predictive accuracy ($R^2 \approx 0.96$) with an empirical sample budget that grows approximately linearly in $N$, in contrast to the combinatorial cost of fully joint profiling. Integrated into a genetic algorithm scheduler, the surrogate scales to an $N=5$ problem with 78,732 feasible joint configurations, remaining within 0.10% of the exhaustive search with zero SLA violations across eight dynamic workload scenarios, while complete online GA decisions take 26 ms median, about $5\times$ faster than exhaustive surrogate search.
