Search papers, labs, and topics across Lattice.
This paper evaluates the effectiveness of three inference optimizations鈥擣used Triton kernels, INT4 quantization, and torch.compile graph breaks鈥攐n Mixture-of-Experts (MoE) models, specifically OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B. Despite promising isolated performance gains, the optimizations fail to deliver significant end-to-end improvements due to inefficiencies in routing and kernel launches, with Fused Triton kernels achieving only 0.999x of the theoretical ceiling. Additionally, the findings reveal that expert substitutability undermines specialization, and that routing fidelity and output quality are distinct objectives, highlighting critical trade-offs in MoE model optimization.
Fused Triton kernels may promise up to 9x speedup, but in practice, they deliver virtually no end-to-end gain due to routing inefficiencies.
Mixture-of-Experts (MoE) models route each token to a few of many expert networks, and that routing is data- dependent in a way standard inference optimizations do not ex- pect. This paper measures what three of them actually deliver on OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B. Fused Triton kernels reach 5.6x to 9.0x in isolation but 0.999x end to end against a measured 1.07x ceiling, because the model spends its time waiting on roughly a thousand kernel launches per forward pass rather than on the arithmetic those kernels improve. INT4 quantization changes on average 0.53 of the eight selected experts per token position, yet replaying exactly those changed routes through full-precision weights reproduces only 2.7% of the quality loss, which makes the experts substitutable rather than specialized. Removing all 23 torch.compile graph breaks, the step prior work treats as the structural fix, makes the model three times slower. A fourth result ties the three together: leaving the routers in FP16 lowers drift by 20% while raising loss, so routing fidelity and output quality are separable objectives. Every number recomputes from committed per-token route dumps.
