Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the challenge of learning in the weight space of transformer-based language models by developing symmetry-aware graph metanetworks. It formalizes the weight-space symmetry group of decoder-only LMs, including continuous general-linear symmetries of attention matrices and permutations of heads, and proposes a graph construction invariant to this group. Experiments on novel transformer-population datasets demonstrate superior performance in tasks like training-corpus prediction and fine-tuning detection compared to existing baselines.
Transformer weight-space learning is now invariant to the full, rich symmetry group of attention heads and token embeddings.
Weight-space learning treats the weights of neural networks as data, processed by other models called metanetworks. A key challenge is that different weights can realize the same function, related by a group of symmetries the metanetwork should respect. While symmetry-aware metanetworks for simple architectures such as MLPs are well developed, a comprehensive treatment of transformers, and in particular transformer-based language models, is still missing. Importantly, their weight space has a richer symmetry group: beyond neuron permutations, attention introduces continuous general-linear symmetries of the attention matrices and permutations of entire heads. Existing methods for transformer weights either capture only part of these symmetries or are restricted to a single, fixed architecture, and none accounts for language-model-specific components such as the token embeddings. Building on graph metanetworks, we propose a graph-based approach that is both symmetry-aware and applicable across depths and widths. We formalize a weight-space symmetry group of complete decoder-only language models and provide a graph construction that is invariant to it. We further introduce two new transformer-population datasets with novel tasks, such as training-corpus prediction and fine-tuning detection, on which our approach notably outperforms existing baselines.