Search papers, labs, and topics across Lattice.
This paper introduces RepBench, a novel benchmark-grounded data layer designed for capability-aligned representation probing in large language models (LLMs). By compiling 13,427 benchmark papers and 353 public datasets, the authors identify 182 capability clusters across 94 capabilities, revealing that traditional evaluation methods may obscure true model capabilities due to their reliance on synthetic data. The findings indicate that while benchmark-pooled capability vectors reveal a more coherent clustering structure, evaluation methods significantly impact the assessment of model performance, highlighting the need for robust evaluation frameworks.
Traditional evaluation methods may misrepresent LLM capabilities, but RepBench reveals a more nuanced understanding through a multi-benchmark approach that uncovers 182 capability clusters.
Representation engineering reads and steers capability directions in large language models, yet methods are typically evaluated on paper-specific synthetic data. The resulting measurements are difficult to compare or reproduce and may reflect surface patterns rather than capabilities. We present RepBench, a benchmark-grounded data layer for capability-aligned representation probing. Crawling 13,427 benchmark papers yields a taxonomy of 182 capability clusters in 13 families; harvesting 353 public benchmark datasets yields 46,149 audited probe texts covering 94 capabilities, each supported by at least two independent benchmarks. This multi-benchmark design reduces dependence on any single source: raw per-text vectors exhibit no natural cluster granularity, whereas benchmark-pooled capability vectors show an interior clustering optimum at a small number of clusters on all 12 evaluated models, with low agreement to the human taxonomy. Under cross-benchmark transfer evaluation across twelve models completed by all four readouts, difference-in-means attains the highest model-level mean on ten models, while logistic regression wins the most capability-model cells. This disagreement shows that the readout method and aggregation criterion are meaningful evaluation dimensions. The pipeline, corpus, and evaluation code are released as a reusable closed-loop workflow.
