Search papers, labs, and topics across Lattice.
Cornserve is introduced as a distributed serving system designed for Any-to-Any multimodal models, addressing the challenges of varying computational paths and scaling requirements across different input/output modalities. It employs a flexible task abstraction for expressing model computation graphs, enabling component disaggregation and independent scaling. The system's record-and-replay execution model and direct tensor data forwarding achieve up to 3.81x higher throughput and 5.79x lower tail latency compared to existing solutions.
Cornserve unlocks significantly faster and more efficient serving of complex multimodal AI models by intelligently distributing computation and data flow.
Any-to-Any models are an emerging class of multimodal models that accept combinations of multimodal data (e.g., text, image, video, audio) as input and generate them as output. Serving these models are challenging; different requests with different input and output modalities traverse different paths through the model computation graph, and each component of the model have different scaling characteristics. We present Cornserve, a distributed serving system for generic Any-to-Any models. Cornserve provides a flexible task abstraction for expressing Any-to-Any model computation graphs, enabling component disaggregation and independent scaling. The distributed runtime dispatches compute to the data plane via an efficient record-and-replay execution model that keeps track of data dependencies, and forwards tensor data between components directly from the producer to the consumer. Built on Kubernetes with approximately 23K new lines of Python, Cornserve supports diverse Any-to-Any models and delivers up to 3.81$\times$ higher throughput and 5.79$\times$ lower tail latency. Cornserve is open-source, and the demo video is available on YouTube.
