Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors reverse-engineered the final MLP layer of GPT-2 Small, revealing a three-tiered exception handling mechanism composed of 27 neurons that route information from ~3,040 residual stream neurons. They identified distinct roles for different neuron groups, including Core neurons that reset vocabulary, Differentiators that suppress incorrect candidates, Specialists that detect structural boundaries, and Consensus neurons that monitor linguistic dimensions. Experiments demonstrate that "knowledge neurons" in layer 11 act as routing infrastructure, amplifying or suppressing signals from attention based on contextual constraint, and that the exception handler operates at the token level.
GPT-2's final MLP layer isn't a black box, but a surprisingly legible routing program with named neurons handling exceptions and scaling attention signals based on context.
The final MLP of GPT-2 Small exhibits a fully legible routing program -- 27 named neurons organized into a three-tier exception handler -- while the knowledge it routes remains entangled across ~3,040 residual neurons. We decompose all 3,072 neurons (to numerical precision) into: 5 fused Core neurons that reset vocabulary toward function words, 10 Differentiators that suppress wrong candidates, 5 Specialists that detect structural boundaries, and 7 Consensus neurons that each monitor a distinct linguistic dimension. The consensus-exception crossover -- where MLP intervention shifts from helpful to harmful -- is statistically sharp (bootstrap 95% CIs exclude zero at all consensus levels; crossover between 4/7 and 5/7). Three experiments show that "knowledge neurons" (Dai et al., 2022), at L11 of this model, function as routing infrastructure rather than fact storage: the MLP amplifies or suppresses signals already present in the residual stream from attention, scaling with contextual constraint. A garden-path experiment reveals a reversed garden-path effect -- GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure. This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11. Code and data: https://github.com/pbalogh/transparent-gpt2