Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
3
0
3
GPT-2's final MLP layer isn't a black box, but a surprisingly legible routing program with named neurons handling exceptions and scaling attention signals based on context.
Transformers waste up to 56% of their MLP compute on near-linear operations, and selectively replacing nonlinear layers with linear ones can actually *improve* performance.
Certain transformer attention heads act like surprisingly robust Bloom filters, remembering which tokens appeared earlier in the context with impressive accuracy and generalizing beyond just repeated names.