Search papers, labs, and topics across Lattice.
This paper introduces Messier, a comprehensive corpus comprising 957,253 records across 30 benchmarks and multiple agent types, aimed at addressing the challenges of evaluating AI agents in interactive environments. By standardizing records based on various criteria, the authors reveal significant disparities in agent performance across different benchmark types, highlighting that while "programming" tasks show rapid improvement, "enterprise workflows" remain particularly difficult. Additionally, the study uncovers that traditional aggregation methods can misrepresent agent capabilities, providing a new framework for more accurate evaluation and capability scaling.
Standardized evaluations reveal that while some AI capabilities are advancing rapidly, others are stagnating, challenging our assumptions about agent performance across domains.
Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules. Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable. We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, with SOC/NAICS classifications for occupational and industry analysis. Using this corpus, we show frontier progress is uneven across benchmark types, with "function calling" saturated, "programming" improving the fastest, and "enterprise workflows" remaining the most challenging. Furthermore, counterfactual rescoring shows that strict all-pass aggregation in multi-verifier tasks can obscure progress and artificially alter agent rankings. From these standardized records, we derive capability scales that align with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.81 and can be specialized by domain, occupation, action space, or verifier type. Messier provides a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures.
