Search papers, labs, and topics across Lattice.
This paper conducts a geometric and stochastic analysis of discontinuities in Sparse Mixture-of-Experts (SMoE) architectures, which are critical for scaling large models in language and vision tasks. By classifying discontinuities by order and establishing asymptotic volume estimates, the authors reveal that lower-order discontinuities dominate the input space, making them more likely to influence model outputs. Additionally, they introduce a smoothing mechanism that mitigates the effects of these discontinuities, leading to improved performance in empirical evaluations while maintaining low computational overhead.
Lower-order discontinuities in Sparse Mixture-of-Experts architectures dominate input space, significantly affecting model outputs and performance.
Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networks. However, this very Top-$k$ expert selection that enables conditional routing also renders the SMoE map inherently discontinuous. In the vicinity of these discontinuity surfaces, even inputs that are arbitrarily close may activate substantially different sets of experts resulting in significantly different outputs. In this work we give a rigorous geometric and stochastic analysis of these discontinuities. We first classify them by order, determined by the number of tied experts at a switching event. Using measure-theoretic slicing arguments, we establish asymptotic volume estimates for the thickened discontinuity surfaces, showing that lower-order discontinuity sets dominate, whereas higher-order ones occupy a vanishingly small relative volume. Next, modeling random perturbations in the input space via a diffusion process, we prove that the path eventually encounter a discontinuity, and moreover that the first hit almost surely occurs on an order-1 discontinuity with explicit finite-time probability bounds. We further derive occupation-time bounds that quantify the duration the random path spend in the neighborhoods of each discontinuity order. These theoretical results imply that inputs are more likely to lie near lower order discontinuities. Motivated by this insight, we propose a simple smoothing mechanism that can be directly applied to existing SMoEs, softly incorporating experts near discontinuities; our analysis guarantees that the added computational overhead remains small while providing localized smoothing near discontinuities, and experiments across language and vision tasks show that smoothing not only enforces continuity of the SMoE map but also enhances empirical performance.
