Search papers, labs, and topics across Lattice.
This paper applies factor analysis to 13,251 evaluation scores from 1,618 language models across 456 benchmarks to investigate the structure of machine intelligence. Findings reveal that while a general intelligence factor explains a significant portion of variance (up to 70.8%), it is not well-proxied by standard intelligence benchmarks, and content-similar tasks do not reliably cluster. This challenges the assumption of a unified, domain-free intelligence construct and suggests that targeting a single conceptual ability is not a supported strategy due to the idiosyncratic nature of first-order abilities.
Machine intelligence is not a singular, domain-free construct, with a general factor explaining at most 70.8% of variance and failing to correlate with standard "intelligence" benchmarks.
A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like fluid intelligence in humans. This assumption is rarely tested directly, and prior attempts have done so only at a much smaller scale. We take a latent variable approach to intelligence in language models, similar to how psychometricians study psychological constructs. Performance in every specific problem set is influenced by a domain-specific and a domain-agnostic latent factor. Using factor analysis as a dimension-reduction technique, we analyzed 13,251 published evaluation scores covering 1,618 language models across 456 different text-only benchmarks. Due to the super-sparse nature of the dataset, we triangulate our analysis across different data densifiers and imputation methods. A robust pattern across different modes of bias is that 1. A general intelligence factor accounts for 70.8% of variance in model performance at our most generous estimate, and far less than that in most of our solutions, 2. Content-similar benchmarks do not necessarily cluster together, and 3. The $g$ factor is not dominated by any common theme, and there is a lack of evidence that it is well-proxied by standard"intelligence"benchmarks. Our findings go against current endeavors of defining, identifying, and targeting general intelligence as a tangible construct in language model development. This leaves the strategy of targeting a single conceptual ability without support, since the first-order abilities it would have to reach are often partially idiosyncratic and not identifiable in practice.
