Search papers, labs, and topics across Lattice.
This paper introduces Embodied-BenchClaw, an autonomous system designed to streamline the construction of embodied spatial intelligence benchmarks through a five-stage pipeline involving intent blueprinting, data collection, structuring, synthesis, and evaluation. By utilizing a coordinated approach with three specialized agents and an extensible Skill Library, the system enhances the reusability and reliability of benchmarks, addressing the limitations of static, labor-intensive methods. Experimental results demonstrate that Embodied-BenchClaw can create verifiable and maintainable benchmarks across various spatial reasoning tasks with significantly reduced manual effort.
Autonomous benchmark construction can now adapt to evolving AI capabilities, making it easier to evaluate embodied spatial intelligence.
Benchmarks are essential for evaluating embodied spatial intelligence, yet their construction is labor-intensive, hard to reuse, and difficult to maintain. Existing embodied benchmarks are often static and may quickly become saturated as models improve, limiting their ability to distinguish new capabilities. We propose Embodied-BenchClaw, an autonomous agentic system for constructing embodied spatial intelligence benchmarks. Given a user-specified evaluation intent, Embodied-BenchClaw automatically produces a complete and continually updatable benchmark package through a five-stage pipeline: intent blueprinting, data collection, structuring and cleaning, benchmark synthesis, and evaluation reporting. The pipeline is coordinated by three agents for planning, construction, and evaluation. To improve reusability and reliability, Embodied-BenchClaw introduces an extensible Skill Library and process quality control, enabling benchmark construction to be composable, verifiable, and repairable. We instantiate multiple benchmarks covering indoor spatial reasoning, outdoor spatial reasoning, robotic manipulation, quadruped robot navigation, UAV/aerial-view understanding, and static benchmark enhancement. These benchmarks span diverse embodied carriers, data sources, and spatial capabilities. Experiments with human evaluation, judge-based assessment, consistency checks, cost analysis, and ablations show that Embodied-BenchClaw can construct verifiable, executable, maintainable, and diagnostically useful embodied spatial benchmarks with reduced manual effort.
