Search papers, labs, and topics across Lattice.
The paper introduces ButterflyMoE, a novel Mixture-of-Experts (MoE) architecture that achieves sub-linear memory scaling with the number of experts by representing experts as geometric reorientations of a shared, quantized substrate. This approach avoids storing independent weight matrices for each expert, instead leveraging learned rotations applied to a shared ternary prototype. The method demonstrates a 150x memory reduction with 256 experts and negligible accuracy loss on language modeling benchmarks, enabling deployment of MoE models on edge devices.
Forget massive MoE parameter counts: ButterflyMoE slashes memory requirements by 150x using learned rotations of a shared, quantized prototype, without sacrificing accuracy.
Linear memory scaling stores $N$ independent expert weight matrices requiring $\mathcal{O}(N \cdot d^2)$ memory, which exceeds edge devices memory budget. Current compression methods like quantization, pruning and low-rank factorization reduce constant factors but leave the scaling bottleneck unresolved. We introduce ButterflyMoE, a method that treats experts not as independent weight matrices but as geometric reorientations of a unified shared quantized substrate. Diversity among experts arises from viewing different angles of shared capacity, not from redundant storage. By applying learned rotations to a shared ternary prototype, each expert yields $\mathcal{O}(d^2 + N \cdot d \log d)$ memory,sub-linear in the number of experts. The key insight: training these rotations with quantization reduces activation outliers and stabilizes extreme low bit training, where static methods collapse. Across language modeling benchmarks, ButterflyMoE achieves 150$\times$ memory reduction at 256 experts with negligible accuracy loss. ButterflyMoE allows multiple experts to fit on edge-constrained devices showing that geometric parameterization breaks linear scaling.
