Yongzhong Xu

Clustering co-activation statistics may hint at attention-head circuits, but only closure testing can confirm their validity—leading to surprising findings in Mixture-of-Experts models.

Induction circuits form up to 20 times earlier than BOS-attractor heads, revealing a critical separation in the developmental phases of attention mechanisms in language models.

Forget what you thought you knew about how models learn: analyzing loss gradients, not just parameter updates, reveals a hidden order of magnitude increase in the coupling between learned features and parameter space.