Search papers, labs, and topics across Lattice.
This paper introduces R2S-Eval, an innovative evaluation pipeline that enhances robot manipulation policy assessment by integrating real-to-sim calibration with vision-language model (VLM) preference evaluation. By generating rollout videos in a simulator that mirrors real-world conditions, R2S-Eval significantly reduces the need for labor-intensive hardware trials and provides a more nuanced analysis of robot performance through pairwise preference rankings. Experimental results demonstrate that R2S-Eval not only aligns with human evaluators' preferences but also uncovers critical behavior quality differences that traditional binary success metrics overlook.
R2S-Eval reveals that automated evaluation can outperform manual success counting by capturing nuanced differences in robot behavior quality.
Evaluating robot manipulation policies is becoming increasingly important as generalist models, particularly vision-language-action (VLA) models, are deployed on physical robots. However, conventional real-world evaluation remains labor-intensive, unstable, and insufficiently informative. It requires repeated hardware trials, manual scene resets, and continuous operator monitoring, may produce different policy rankings across repeated evaluations, and primarily relies on success-rate metrics that provide limited information about execution quality. In contrast, humans assess robot performance by observing and comparing complete behaviors rather than relying solely on binary success outcomes. To this end, we propose R2S-Eval, an evaluation pipeline that combines real-to-sim calibration with vision-language model (VLM) preference evaluation. The real-to-sim component efficiently generates rollout videos in a simulator calibrated to the real-world evaluation setting, thereby reducing the need for repeated hardware trials. The VLM evaluator assesses the execution quality of rollout videos and produces pairwise preferences, which are subsequently aggregated into policy rankings. We further introduce a protocol to assess whether the proposed evaluation pipeline yields validated policy conclusions while mitigating the key challenges of conventional real-world evaluation. Experiments in both simulation and real-world settings demonstrate that R2S-Eval produces reliable and stable policy conclusions, achieves agreement with human preferences, substantially reduces repeated hardware-operation effort, and reveals behavior-quality differences that are not captured by binary success labels. In general, R2S-Eval advances robot evaluation from manual success counting toward automated, statistically stable, and quality-aware evaluation of robot behavior. Project page: https://r2s-eval.github.io.
