Search papers, labs, and topics across Lattice.
This study investigates how next-token prediction models, despite being trained with one-hot labels, manage to learn latent semantic structures that reflect contextual relationships among words. By analyzing three synthetic settings, the authors demonstrate that semantic geometry emerges early in training, leading to clustering of representations based on shared attributes, even in the absence of explicit supervision. However, this emergent structure is transient, as models eventually transition to a symmetric state where representations are uniformly spaced, aligning with the predictions of Neural Collapse theory.
Language models can uncover latent semantic structures from one-hot training labels, but this emergent geometry is fleeting and ultimately gives way to a uniform representation.
Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs. This creates a puzzle: next-token prediction language models are trained predominantly (as context length increases) with one-hot labels: the same context is very unlikely to appear twice in training with different labels. However, they clearly learn latent structural features. That is, despite the one-hot training regime, a language model's contextual embeddings represent the fact that the next word in ''Mary broke the ___'' is likely to be filled by tokens in the latent classes of a) medium-sized, b) rigid, c) inanimate nouns. How does gradient descent find such categorical semantic structure when co-occurrence statistics collapse to one-hot sparsity, eliminating any shared next-tokens among different contexts? To investigate this tension we identify three synthetic controlled settings where inputs have latent semantic factors but are mapped to distinct one-hot labels. We find that semantic geometry emerges early in training, and that representations cluster by shared attributes despite receiving no explicit supervision to do so. This structure is transient: with sufficient capacity and time, the model eventually reaches the predicted symmetric state where all representations are equally separated. We study this phase transition through Gram matrix analysis and propose a preliminary modification to the commonly used unconstrained features model to capture the emergent semantic geometry.
