Search papers, labs, and topics across Lattice.
This paper details a comprehensive optimization of the inference pipeline for the MiMo-V2.5 model series, integrating Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders. The authors implement a series of engineering innovations, including layerwise prefetching and a high-performance distributed cache system (GCache), achieving significant reductions in storage and computation costs while maintaining high cache hit rates. The resulting system represents a breakthrough in efficiently deploying large-scale LLMs with complex architectures in production environments.
Achieving $O(W)$ storage efficiency and high cache hit rates in a large-scale LLM serving system could redefine performance benchmarks for hybrid architectures in production.
We present a full-pipeline inference optimization for the MiMo-V2.5 model family, which combines Hybrid Sliding Window Attention (Hybrid SWA), sparse Mixture-of-Experts (MoE), and multimodal encoders. While Hybrid SWA can ideally reduce both attention compute and KVCache storage significantly compared to Full Attention, realizing these gains in production requires substantial engineering effort. We systematically optimize the KVCache system with layerwise prefetch, SWA-aware prefix cache trees, and specialized placement strategies, achieving strict $O(W)$ SWA storage and high cache hit rates. We further build GCache, a high-performance distributed cache infrastructure with RDMA-optimized networking, and develop a KVCache-affinity router to reduce computation while preserving load balancing. We also optimize for multimodal inputs, including GPU image preprocessing, parallel video decoding, and multimodal cache sharing. Together, these optimizations constitute the first large-scale LLM serving system in production that efficiently covers the Hybrid SWA + MoE + multimodal composite architecture.
