Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FMplex, a model-serving system that virtualizes foundation models (FMs) to optimize resource usage across multiple downstream tasks. By creating virtual foundation models (vFMs) that allow for task-specific customizations while sharing a single physical FM, FMplex significantly reduces memory waste and improves efficiency in model deployment. The results demonstrate that FMplex can lower latency by up to 80% compared to traditional spatial partitioning methods and support up to six times more tasks simultaneously at cluster scale.
FMplex slashes latency by up to 80% while enabling six times more tasks to share a single foundation model, revolutionizing model serving efficiency.
Foundation models (FMs) are increasingly used as backbones for downstream tasks across language, vision, time-series, and multimodal applications. Yet existing model-serving systems deploy each customized task as an independent model instance, thereby replicating heavyweight backbones, wasting accelerator memory, and losing opportunities to amortize batching and loading costs. This paper presents FMplex, a serving system that treats FM backbones as a virtualization substrate for deployment sharing. FMplex presents each task with a virtual foundation model (vFM), a logically private FM instance backed by a shared physical FM. This abstraction lets independently customized tasks share a backbone while preserving task-specific extensions, independent lifecycles, and task-level isolation. In addition, we propose a batch-aware fair-queueing scheduler that combines weighted task-level sharing with inter- and intra-task batching across colocated tasks. We implement a FMplex-based serving stack spanning task construction, sharing-aware deployment, and runtime execution. Across 7 FM backbones (16 variants) and 92 downstream tasks, FMplex reduces latency by up to 80% over spatial partitioning and 33.3% over best-effort co-location, while hosting up to 6x more tasks at cluster scale.