Search papers, labs, and topics across Lattice.
This paper introduces Trident, an adaptive scheduling framework designed to optimize the throughput of heterogeneous multimodal data pipelines common in modern AI infrastructure. Trident uses Gaussian Process regression with anomaly filtering to estimate per-operator throughput, and employs memory-constrained Bayesian optimization to determine OOM-safe configurations in response to workload shifts. The framework then leverages a mixed-integer linear program to jointly optimize operator parallelism, placement, and configuration transitions, resulting in significant throughput improvements compared to static baselines.
Squeezing up to 2x more throughput from multimodal data pipelines is now possible with Trident, a new adaptive scheduler that dynamically optimizes operator parallelism and placement while avoiding OOM errors.
The rapid adoption of large language models and multimodal foundation models has made multimodal data preparation pipelines critical AI infrastructure. These pipelines interleave CPU-heavy preprocessing with accelerator-backed (GPU/NPU/TPU) inference and produce massive intermediate artifacts. Achieving high throughput is difficult because workloads are highly non-stationary: regime shifts, input-dependent inference, and transient memory spikes cause rapid performance fluctuations and out-of-memory (OOM) failures. Existing schedulers typically rely on threshold-based autoscaling or assume synchronous, homogeneous operators, leading to poor efficiency. We present Trident, an adaptive scheduling framework for heterogeneous multimodal pipelines on fixed-resource clusters. Trident closes the loop across three coupled layers: (i) an observation layer that estimates per-operator sustainable throughput for asynchronous operators via Gaussian Process regression with anomaly filtering; (ii) an adaptation layer that detects workload shifts online and performs memory-constrained Bayesian optimization to recommend OOM-safe configurations; and (iii) a scheduling layer that solves a mixed-integer linear program to jointly optimize operator parallelism, placement, and configuration transitions under heterogeneous compute and bandwidth constraints, accounting for cold-start overhead via rolling updates. Decisions trigger sample invalidation and model refresh to keep estimates consistent with the active configuration. Implemented on Ray Data, Trident improves end-to-end throughput by up to 2.01x on a document curation (PDF) pipeline and 1.88x on a video curation pipeline over a static baseline, with low overhead suitable for online re-optimization.
