Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces parameter-trajectory crosscoding to directly track the evolution of the unembedding matrix during language model pretraining, revealing that task information can be encoded in hidden states before the readout matrix expresses it. By analyzing Pythia's pretraining, the authors pinpoint a narrow window where the readout reorganizes and WordNet categories become decodable, while linear probes on hidden states already capture this information. Manipulating the readout's learning rate demonstrates that the speed of readout development directly influences the observed gap between internal encoding and external expression of task knowledge.
Language models encode task information in hidden states *before* their unembedding matrix can express it, suggesting current developmental timelines are misleading.
When does a language model acquire a capability? Developmental studies usually answer this by tracking when models start producing the correct token. But that prediction passes through the unembedding matrix, or readout, which is itself still being learned during pretraining. We show that a model's hidden states can encode task information before its own readout expresses it, so timelines read from outputs conflate what the model encodes with what its readout expresses. To track the readout directly, we introduce parameter-trajectory crosscoding, which fits one sparse dictionary to every token's readout row at every checkpoint. Unlike crosscoders on activations, it needs only checkpoint weights. We use it to locate a narrow early window, 0.4-1.1% of the way through Pythia's pretraining, in which the readout reorganizes and WordNet word categories become decodable from its rows. By then, linear probes on the hidden states already decode task information that the model's own outputs do not yet show. To test whether such gaps depend on how fast the readout learns, we pretrain small models from scratch that differ only in the readout's learning rate. We find that raising it makes the readout develop earlier and the gap close sooner, with little effect on final loss. Together, these results imply that early in training, where many developmental transitions are dated, a behavioral date can reflect when the readout learns to express task information and not when the model learns it.