Search papers, labs, and topics across Lattice.
This paper introduces RoboLab, a high-fidelity simulation benchmarking framework designed to evaluate the generalization capabilities of task-generalist robotic policies. By creating the RoboLab-120 benchmark, which features 120 diverse tasks across three competency axes, the authors address the limitations of existing benchmarks that fail to capture true generalization due to domain overlap. The results reveal significant performance gaps in current state-of-the-art models, highlighting the framework's potential to provide deeper insights into policy robustness and sensitivity to external factors.
RoboLab exposes critical performance gaps in leading robotic models, revealing that high-fidelity simulations can better assess generalization than traditional benchmarks.
The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing. Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness. We introduce RoboLab, a simulation benchmarking framework designed to address these challenges. Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which external factors most strongly affect that behavior under controlled perturbations. First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot- and policy-agnostic manner within a physically realistic and photorealistic simulation. With this, we propose the RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational competency, across three difficulty levels. Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, indicating that high-fidelity simulation can serve as a proxy for analyzing performance and its dependence on external factors. Evaluation with RoboLab exposes significant performance gap in current state-of-the-art models. By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies.
NVIDIA