Search papers, labs, and topics across Lattice.
This study investigates the impact of route-block interventions on packed quantized matrix multiplication in a mixture-of-experts (MoE) framework, specifically within a Qwen3-Coder AWQ layer-6 fixture. By manipulating token alignments and routes, the authors demonstrate that these interventions can select specific arithmetic trajectories, revealing the intricate relationships between routing decisions and output behaviors. The findings indicate that even minor changes in routing can lead to significant shifts in model performance, emphasizing the importance of preprocessing in MoE inference.
Route-block interventions can drastically alter model outputs, revealing hidden dependencies in expert alignment that challenge conventional assumptions about MoE preprocessing.
Mixture-of-experts (MoE) inference first aligns routed tokens into padded expert blocks, then executes packed quantized matrix multiplication over those blocks. This preprocessing is often treated as bookkeeping. In one pre-specified Qwen3-Coder AWQ layer-6 fixture on a pinned vLLM/Marlin build and RTX 3090 runtime, we show that the tested route-block interventions select exact packed arithmetic trajectories. Two fixed preconstruction histories produced distinct native alignments and exact trajectories. Injecting the opposite alignment transferred W13, activation, routed-W2, and final outputs. Permuting two routes within one block preserved each native trajectory, while exchanging two prior-data-selected routes across the boundary between expert-106 blocks 40 and 41 transferred the complete opposite trajectory. Source- and binary-derived schedule geometry maps those blocks to direct/full-K and split/global-reduction classes. Forcing a single-slice 200-block grid made W13 bitwise equal. Stable canonical construction made both histories converge to a third exact trajectory. The confirmatory cohort contains 70 valid cold processes and seven required perturbation rejections. This is a causal mechanism result for one fixture, not a prevalence, allocator, portability, or serving-impact claim.
