Search papers, labs, and topics across Lattice.
This paper investigates whether the linear recoverability of world properties (geographic and temporal) from LLM hidden states truly indicates world-like internal representations. The authors demonstrate that similar spatial and temporal information can be recovered from static co-occurrence-based word embeddings (GloVe and Word2Vec) using ridge regression probes. Analysis reveals that interpretable lexical gradients, such as country names and climate-related vocabulary, are crucial for recovering these signals, suggesting that simple co-occurrence statistics in text encode more world structure than previously thought.
Static word embeddings like GloVe and Word2Vec can achieve surprisingly high accuracy (R^2 up to 0.87) in recovering geographic and temporal information, challenging the interpretation of similar findings in LLMs as evidence of complex world models.
Recent work interprets the linear recoverability of geographic and temporal variables from large language model (LLM) hidden states as evidence for world-like internal representations. We test a simpler possibility: that much of the relevant structure is already latent in text itself. Applying the same class of ridge regression probes to static co-occurrence-based embeddings (GloVe and Word2Vec), we find substantial recoverable geographic signal and weaker but reliable temporal signal, with held-out R^2 values of 0.71-0.87 for city coordinates and 0.48-0.52 for historical birth years. Semantic-neighbor analyses and targeted subspace ablations show that these signals depend strongly on interpretable lexical gradients, especially country names and climate-related vocabulary. These findings suggest that ordinary word co-occurrence preserves richer spatial, temporal, and environmental structure than is often assumed, revealing a remarkable and underappreciated capacity of simple static embeddings to preserve world-shaped structure from text alone. Linear probe recoverability alone therefore does not establish a representational move beyond text.
