Search papers, labs, and topics across Lattice.
This study introduces LLMPEDIA, a tool that materializes and audits approximately 1.3 million articles from the parametric memory of three language model families, revealing significant gaps in their encyclopedic knowledge. The findings indicate that while flagship models score above 90% on traditional benchmarks like MMLU, their actual factual accuracy is only 68.4%, with over 30% of claims deemed insufficient for verification. By providing a browsable and verifiable encyclopedia of claims, LLMPEDIA highlights the limitations of current benchmarks and facilitates deeper exploration of model knowledge and factuality.
Language models may score high on benchmarks, but LLMPEDIA reveals their true factual accuracy is only 68.4%, exposing critical gaps in their encyclopedic knowledge.
Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the experimenter thought to ask, the availability bias of fixed question sets. LLMPEDIA makes this bias measurable and browsable. We recursively materialized ~1.3M articles from three model families' parametric memory (GPT-5-mini, DeepSeek-V3.2, Llama-3.3-70B) without retrieval, then audited a stratified sample of atomic claims against Wikipedia and a curated web stack, coloring every claim supported, refuted, or insufficient (Saeed and Razniewski, 2026). On a uniform random sample the true rate is 68.4% - more than 21 pp below MMLU - with 30.5% of claims insufficient: assertions no benchmark probes and the world's largest encyclopedia cannot adjudicate - long-tail knowledge or plausible hallucination, the evidence cannot tell - extending to free text the coverage gap GPTKB established for triples (Hu et al., 2025). The resulting live, open encyclopedia lets visitors inspect this frontier one claim at a time through five one-click views - link-traversal exploration, claim-level factuality, cross-model and political-persona comparison, and a guided topic drill-down - each page, claim, and verdict at a stable URL. LLMPEDIA is live at https://llmpedia.net
