Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces a simple method using singular value decomposition (SVD) of the `lm_head` weight matrix to reveal interpretable semantic subspaces within LLMs, exposing the model's learned vocabulary clusters and training data composition. Applying this to GPT-OSS-120B, Gemma-2-2B, and Qwen2.5-1.5B reveals distinct spectral and cluster structures, including ethically concerning subspaces that persist even after post-training alignment. They also introduce the Vocabulary Cluster Score (VCS) and Weighted Projection Score (WPS) to quantify subspace coherence and detect glitch tokens, respectively, without requiring model inference.
LLMs are hiding interpretable semantic structures in plain sight: just five lines of code and an SVD can expose their training data secrets and even ethically problematic content.
We show that singular value decomposition of the lm_head} weight matrix of a transformer-based large language model -- requiring only five lines of PyTorch and no model inference -- reveals interpretable semantic subspaces directly from the model weights. Each left singular vector identifies the vocabulary tokens most readily selected when the hidden state aligns with the corresponding singular direction; inspecting these clusters exposes the model's training data composition and curation philosophy. Analysing GPT-OSS-120B, Gemma-2-2B, and Qwen2.5-1.5B, we find that singular value spectra and vocabulary cluster structures differ systematically across models: GPT exhibits a graduated hierarchy of functionally differentiated subspaces; Gemma is dominated by pre-nineteenth-century English orthography, forming a stepwise clustering structure that may contribute to high output controllability; and Qwen exhibits broad multilingual coverage alongside subspaces whose vocabulary the authors have determined to be ethically inappropriate for direct publication. Base-instruct comparison reveals that ethically concerning subspaces originate in pretraining and are not removed by post-training alignment. We introduce the Vocabulary Cluster Score (VCS) to quantify subspace coherence, and the Weighted Projection Score (WPS) as a static glitch token detector; applying WPS to GPT-OSS-120B recovers shokubutsu-hyakka-tsu (ID 137606), a well-known glitch token widely reported in the CJK language community, without any model inference. We propose a taxonomy of root causes for problematic vocabulary content and call for lm_head} SVD analysis to be adopted as a standard pre-release safety auditing step. Our findings further suggest directions toward SVD-guided tokenizer optimisation and more controllable LLM design.