Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
RLE-Bench is introduced to evaluate coding agents' broader engineering capabilities in robotics, moving beyond single-artifact performance to assess tasks like building, integration, diagnosis, and improvement under constraints. The benchmark comprises four robotics development workflows—interactive control, policy learning, perception/estimation, and mechanical design—using diverse metrics to quantify agent performance on trained policies, built harnesses, and designed structures. An aggregated RLE Index and workflow-specific profiles enable systematic comparison of agent capabilities, with case studies further illuminating current strengths and limitations.
Coding agents are failing to demonstrate robust engineering skills in robotics, with RLE-Bench revealing significant gaps in multimodal reasoning and artifact integration beyond simple control.
Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics. Existing robotics benchmarks, however, primarily focus on the performance of individual artifacts, such as policies or controllers, offering limited coverage of coding agents'broader engineering capabilities. Real-world robotics extends beyond control: agents must build, integrate, diagnose, and improve heterogeneous artifacts under resource constraints and reason from multimodal feedback. To evaluate these broader capabilities, we introduce RLE-Bench, a benchmark of robot-learning tasks spanning four representative robotics development workflows: interactive control, policy learning, perception and estimation, and mechanical design. We use diverse task-specific metrics to evaluate the artifacts submitted by the coding agents, from the success rate the agents achieved to the policy agents trained, the harness agent built, and the mechanical structures the agent designed. We aggregate these metrics into an overall RLE Index and report workflow-specific capability profiles, enabling systematic comparison of coding agents'capabilities across multiple capability dimensions. Beyond performance ranks, we also conduct in-depth case studies examining agent behavior on representative tasks, highlighting both current capabilities and limitations, and pointing to the opportunities robotics tasks have to offer for future agent training.