Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the capacity of ~100B parameter language models to represent a decodable map of the night sky from their residual streams. The findings reveal that most open-source models possess this representation, which can be effectively elicited through targeted prompts, achieving high variance scores and low angular error. Notably, this representation is identified as the first instance of a curved high-dimensional irreducible feature manifold, indicating a significant advancement in understanding the internal structures of language models.
Language models can decode a detailed representation of the night sky, revealing a surprising depth in their internal feature structures.
We analyze whether language models of size ~100B have a representation of the night sky map that is decodable from their residual stream. We find that most of the considered open-source models do have such a representation, and it often even surfaces to the top principal components on prompts that ask questions like ``what is close to this object in the night sky''. In all but one model this representation showed significant scores in LOO testing, containing up to 65-85% of variance ($R^2$-score) and having median angular error down to $12^\circ-21^\circ$. We verify that our representation is not a simple leak from a correlated flat representation. To our knowledge, this representation is the first example of a curved high-dimensional irreducible feature manifold. Codes used in the paper are published at https://github.com/l3erdnik/Decodable-sky