Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ArmnetBench v0.1, a benchmark designed for the real-world evaluation of robot manipulation policies using a fleet of low-cost SO-101 robotic arms. It systematically compares seven manipulation policies across twelve tasks, utilizing 2,518 policy rollouts and 600 reference demonstrations, all labeled for success, suboptimal performance, or failure. The findings not only validate the arm farm setup but also provide high-quality trajectories that can enhance downstream learning tasks, such as training reward and predictive models on mixed-quality data.
Real-world evaluation of robot manipulation policies just got a major upgrade with ArmnetBench v0.1, enabling robust comparisons across diverse tasks and policies.
Real-world evaluation is a bottleneck in developing generalist robot manipulation policies. Each rollout requires physical hardware and an operator to set up, reset, and score it. We introduce ArmnetBench v0.1, a benchmark run on a fleet of low-cost SO-101 cells under light on-site supervision. v0.1 validates this arm farm end to end and compares 7 policies across 12 tasks with both single-arm and bimanual configurations. Each policy is trained or fine-tuned on 50 demonstrations per task; the benchmark contains 2,518 policy rollouts and 600 reference demonstrations. All 3,118 episodes carry a three-way label (successful, suboptimal, or failure). Policy rollouts are human-scored, while demonstrations are successful by construction. Beyond evaluation, its quality-labelled trajectories support downstream learning, from reward and predictive world models to policies trained on mixed-quality data. The leaderboard is an initial comparison under this shared budget. We release the 3,118 core episodes in LeRobot v3.0 and RoboMeter formats.