Search papers, labs, and topics across Lattice.
Omni-Flow is a distributed scheduling framework designed to address the challenges of multimodal inference in large language models (LLMs) by providing a unified orchestration of workflows, efficient data transmission, and shared key-value (KV) caching. The framework employs a three-layer abstraction鈥擟ontrol Flow, Data Flow, and Compute Flow鈥攖o streamline the orchestration of heterogeneous computing units and facilitate low-latency data sharing across diverse roles. Experimental results demonstrate that Omni-Flow effectively supports complex multimodal tasks, such as omni-modal dialogue and intricate image generation, while maintaining a consistent programming model.
Omni-Flow revolutionizes multimodal inference by integrating flexible orchestration, efficient data sharing, and KV cache reuse into a single cohesive framework.
As large language model (LLM) inference evolves from text-only to multimodal paradigms, inference systems face three challenges: (1) flexible orchestration of multimodal workflows, where heterogeneous computing units exhibit complex dependencies and concurrent control; (2) efficient transmission of massive intermediate data across processes and nodes, with tensors flowing at high speed among heterogeneous roles; and (3) efficient sharing of KV caches and model weights across roles to eliminate redundant GPU memory. Existing solutions deploy LLMs and diffusion models independently, lacking a system-level abstraction for multimodal pipelines; this scatters orchestration logic, tightly couples transmission paths to specific models, and incurs high cost to integrate new models. To address these challenges, we present Omni-Flow, a distributed scheduling framework for multimodal inference through a three-layer abstraction. The Control Flow layer defines workflows via a Python DSL, orchestrating heterogeneous units into a unified dataflow graph that supports static DAGs and dynamic routing, with built-in service discovery and diverse load-balancing strategies. The Data Flow layer provides a distributed KV cache abstraction beyond prefill/decode separation, unifying allocation and enabling direct cross-role transmission across a three-tier paged storage hierarchy (GPU/CPU/SSD) over zero-copy, low-latency channels. The Compute Flow layer supports complex multimodal prefix matching for KV reuse across multi-turn dialogues, and takes over KV cache and sampling logic via a unified SGLang interface, letting diffusion models directly reuse the LLM forward path under unified parallel semantics. We demonstrate that Omni-Flow supports diverse heterogeneous scenarios with a consistent programming model, including omni-modal dialogue (LongCat-Next) and complex image generation pipelines (HunyuanImage-3).
