Search papers, labs, and topics across Lattice.
This paper introduces RaMP, a runtime-aware dispatch framework for Mixture-of-Experts (MoE) inference that optimizes kernel configuration based on both batch size and expert routing distribution. RaMP uses a performance-region analysis derived from hardware constants to predict the effectiveness of different optimizations, achieving high accuracy across diverse architectures. By selecting the fastest kernel configuration using a four-parameter wave cost model based on the runtime expert histogram, RaMP achieves significant speedups in MoE inference compared to static dispatch methods.
Stop leaving 10-70% of your MoE kernel throughput on the table: RaMP dynamically optimizes kernel configuration based on runtime expert routing, achieving up to 1.41x end-to-end speedup in vLLM serving.
The optimal kernel configuration for Mixture-of-Experts (MoE) inference depends on both batch size and the expert routing distribution, yet production systems dispatch from batch size alone, leaving 10-70% of kernel throughput unrealized. We present RaMP, a routing-aware dispatch framework. A performance-region analysis derives, from hardware constants alone, when each optimization helps, correctly predicting all 8 tested architectures, including 3 unseen. A four-parameter wave cost model selects the fastest configuration from the runtime expert histogram, achieving 0.93% mean regret versus exhaustive search, fitted from just 10-24 minutes of one-time profiling per model. Because the model depends only on CTA grid geometry, it is kernel-agnostic: applied to Alpha-MoE, it delivers 1.14x with no source modification. Paired with a co-designed CuTe DSL kernel exposing 134-268 polymorphic configurations, RaMP delivers 1.22x kernel speedup over static dispatch and 1.30x end-to-end speedup in vLLM serving over Triton, 1.41x over DeepGEMM, and 1.13x over FlashInfer CUTLASS.
