Search papers, labs, and topics across Lattice.
This paper extends attention-only dynamical theories of Transformers by incorporating the feed-forward network (FFN) as a local steering field that influences token states in residual direction space. The authors demonstrate that the tangential component of the FFN is crucial for effective motion in this space, revealing that critical residual directions align with nonlinear projective equilibria and are influenced by commutator defects. Experimental results across various models show that leveraging the tangential FFN component significantly enhances angular prediction accuracy while maintaining model quality and output diversity, particularly as model size increases.
Retaining only the tangential component of the feed-forward network can preserve model quality while enhancing prediction accuracy in Transformer dynamics.
Attention-only dynamical theories model Transformer residual directions as particles aggregating on a sphere. We extend this framework by incorporating the feed-forward network (FFN) term as a local steering field acting on each token state. The resulting theory predicts that the tangential component of the FFN field is necessary for motion in residual-direction space, that critical residual directions correspond to nonlinear projective equilibria, and that a commutator defect determines when a finite attention--FFN block can be accurately approximated by a parallel, additive flow. Across GPT-2, Pythia, Mistral, and Llama models, the extended theory improves one-step angular prediction relative to an attention-only baseline, with the contribution of the FFN increasing from GPT-2 to Llama-3-8B. Intervention experiments show that retaining only the tangential FFN component preserves most model quality, whereas retaining only the radial component causes performance to collapse. The tangential component also preserves output diversity under aggregation pressure. As a practical application, layers with small commutator defects can be approximately parallelized with only a modest increase in loss, whereas layers with large defects degrade rapidly. These findings support the interpretation of FFN layers as directional steering fields that shape Transformer residual geometry and govern the feasibility of block-level interventions.
