Search papers, labs, and topics across Lattice.
This study constructs a comprehensive matrix of 51 physical AI models evaluated across 12 benchmarks to investigate the redundancy in benchmark reporting. The analysis reveals significant redundancy, with 22 out of 51 models shifting their rankings by three or more places when redundant benchmarks are collapsed. By selecting a more efficient subset of benchmarks, the authors achieve 78.5% of the original utility, suggesting a streamlined approach to benchmarking that could enhance model evaluation consistency.
Redundant benchmarks can drastically alter model rankings, with 22 models shifting positions significantly when redundancy is addressed.
Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse and the relationship between benchmarks unmeasured. We construct a matrix of 51 models on 12 physical AI benchmarks, selected from a registry of 51 benchmarks and 152 models by reporting density, combining scores from model cards and benchmark papers with our own evaluation runs under each benchmark's official protocol. We measure how much information the benchmarks share and show quantitative evidence of Redundancy. Redundancy affects reported rankings: collapsing the two substitute pairs into single columns moves 22 of 51 models by three or more places under an equally weighted average. We then select benchmarks greedily under a utility combining score dispersion with variance not explained by the already-selected set, and obtain a four-benchmark subset retaining 78.5\% of the utility of all 12, on which we fit a Bradley--Terry ranking. The procedure requires only benchmark-level scores with sufficient overlap and is not specific to physical AI.
