Search papers, labs, and topics across Lattice.
This study investigates the phenomenon of routing divergence in Mixture-of-Experts (MoE) architectures, where identical weights can lead to different expert routing for the same token. By analyzing the contributions of routing and content terms in a self-distillation context, the authors reveal that the routing term has a minimal impact on output, accounting for only 1.6 times the block output, while the content term exhibits a strong directional influence. The findings challenge the assumption that routing changes imply behavioral influence, emphasizing the need to measure exposure before attributing effects to routing decisions.
Routing changes in MoE models may not influence behavior as expected, with the routing term contributing less than half of the natural context effect.
Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1.6\times$ as a fraction of block output, while its residual-stream exposure spans $3.2\times$. Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.
