Search papers, labs, and topics across Lattice.
This paper surveys the evolution of embodied benchmark construction, highlighting a five-stage pipeline that transitions from manual curation to automated processes enhanced by foundation models and agentic workflows. It reveals that while automation can lower initial benchmark costs, it often reallocates expenses towards validation, auditability, and governance, which are critical for maintaining reliability in evaluations. The findings emphasize the need for benchmarks that are not only expansive but also manageable and accountable over time to ensure effective assessment of embodied intelligence systems.
Automation in benchmark construction shifts costs from creation to governance, revealing a hidden complexity in evaluating embodied intelligence.
Embodied intelligence now spans navigation, household assistance, manipulation, autonomous driving, aerial agents, and multimodal large-model control. This expansion has made benchmark construction a central bottleneck for reliable evaluation. Unlike static datasets, embodied benchmarks combine task specifications, environments, robot data, demonstrations, annotations, metrics, evaluation scripts, and release policies into a single evaluation system. This survey reviews the literature through a five-stage construction pipeline: requirement and task construction, data acquisition, data cleaning and annotation, benchmark suite generation and metric definition, and evaluation execution with diagnostic feedback. For each stage, the survey analyzes the transition from manual curation to traditional automation, foundation-model assistance, and agentic closed-loop workflows. It also compares qualitative construction costs across human labor, data and asset acquisition, compute and simulation, validation and debugging, governance and maintenance, and rework risk. The main conclusion is that automation does not simply reduce benchmark cost. Instead, it often shifts cost toward validation, auditability, version control, and long-term governance. Progress in embodied evaluation will therefore depend not only on larger benchmark suites, but also on construction pipelines that are diagnosable, auditable, and responsibly refreshable.
Tsinghua AI