Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel axiomatic framework for evaluating latent thought representations in large language models (LLMs), focusing on metrics that reveal representational shortcomings often obscured by benchmark performance. By formalizing four axioms—Causality, Minimality, Separability, and Stability—and applying them to audit open-weight LLMs across 23 reasoning tasks, the authors demonstrate that no model satisfies all axioms simultaneously. The findings indicate a structural gap in representation quality, as models fail to distinguish between questions within the same task and encode minimal additional information beyond input embeddings.
No existing LLM meets all four critical axioms of thought representation, revealing a fundamental flaw in how we evaluate model reasoning capabilities.
We introduce an axiomatic evaluation framework for latent thought representations in LLMs, comprising metrics that are independent of downstream benchmark scores and reveal representational failures that benchmark accuracy masks. Existing evaluations conflate representation quality with model capacity. Therefore, failures cannot be attributed to the representation rather than to the model that processes it. We formalize four functional axioms (Causality, Minimality, Separability, and Stability) and define a quantitative measure for each, computed directly on the representation independently of downstream accuracy. We audit open-weight LLMs across 23 reasoning tasks (e.g., Spatial Reasoning, Factual QA). We find that no candidate satisfies all four axioms simultaneously, that the representations distinguish task type reliably but cannot distinguish between two questions within the same task, and that the representations encode little information beyond what is already present in the input embedding. The failure is consistent across dense, reasoning-distilled, and RL-trained model families, indicating that the gap is structural rather than a property of model size or training procedure.