Search papers, labs, and topics across Lattice.
This study investigates the interpretability of Block Attention Residuals (Block AttnRes) by analyzing routing mechanisms in two models with identical architecture but different training paradigms. The authors find that while architectural exposure allows for direct observation of information flow, it does not guarantee mechanistic understanding, as evidenced by the presence of distinct routing motifs in the trained model that do not correlate with causal contributions. Ultimately, the research highlights that structured routing only emerges when it is integrated into the training process, necessitating further causal interventions to validate interpretive claims.
Architectural exposure of routing in neural networks reveals surprising dissociations between routing mass and causal importance, challenging assumptions about interpretability.
Block Attention Residuals (Block AttnRes) by replace fixed additive residuals with a learned softmax over earlier depth-source representations, surfacing cross-layer routing as an inspectable tensor in the forward pass. This is a tempting interpretability target: information flow normally inferred indirectly is now directly observable. We ask whether such exposure suffices for mechanistic interpretation. We probe two same-scale ($0.6$B) Block AttnRes checkpoints under identical routing-ablation interventions: a vanilla Qwen3 inference-wrapped through a deterministic recency-bias schedule that the codebase admits as a routing-equivalent loading path, and a Block AttnRes Qwen3 trained from scratch with routing as part of optimisation. The wrapped baseline's routing weights are content-independent and reproduce the schedule's analytic prediction. The trained AttnRes checkpoint instead exhibits three localised routing motifs: an embedding-source pathway through early-layer MLP, a current-state pathway through early-layer attention and MLP, and an older-history pathway through late-layer attention. Beyond this stratification, we find a sharp dissociation between average routing mass and causal importance: in both sublayers, the largest mass slice is not the largest causal contribution, and one source family carries appreciable mass with no detectable causal role under intervention. Architectural exposure of routing is therefore necessary but not sufficient for mechanistic interpretation: structured depth routing emerges only when routing has been part of training, and even then, descriptive routing summaries should be treated as candidate hypotheses to be tested by causal interventions, not as evidence of mechanism in their own right.
