Search papers, labs, and topics across Lattice.
This study investigates the inter-layer dependencies of neurons within the feedforward network (FFN) blocks of Transformer architectures, revealing that the activation of a neuron can be effectively explained by a sparse subset of preceding neuron activations and attention outputs. A novel training-free attribution method was introduced to quantify the influence of upstream neurons on target neuron activations, demonstrating that significant activation fidelity can be achieved even when most inputs are masked. The findings indicate that FFNs, despite their dense parameterization, exhibit structured and sparse dependencies, which could enhance interpretability and efficiency in model inference.
Sparse neuron activations in Transformers reveal that only a few upstream signals are crucial for maintaining activation fidelity, challenging assumptions about dense parameter reliance.
Feedforward network (FFN) blocks account for a large fraction of the parameters and computation in Transformer architectures, yet their internal structure remains difficult to interpret due to the additive superposition induced by the residual stream. We examine whether the activation of an FFN neuron can be explained by a sparse set of preceding neuron activations and attention outputs. We introduce a training-free attribution method that estimates the relative influence of upstream neurons and attention outputs on a target neuron's activation. Empirically, across models and layers, we find that small subsets of preceding activations and attention outputs suffice to preserve neuron activations with high fidelity when all remaining inputs are masked with their average values. Effective sparsity is even greater when accounting for the inherent activation sparsity of upstream layers. Moreover, applying the neuron-specific masks in all layers simultaneously, such that the induced deviations propagate through the network, leaves model perplexity largely unchanged at moderate sparsity levels. These results demonstrate that, despite dense parameterization, FFNs exhibit sparse and structured inter-layer dependencies at the neuron level. Our method provides a practical, scalable tool for circuit-level interpretability and identifies candidate sparse pathways with potential implications for efficient inference.
