Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the relationship between demonstration-curation metrics and the effectiveness of behavior-cloning policies in training on curated data, specifically within a LIBERO pick-and-place benchmark. The findings reveal a stark decoupling between defect detection performance and the resulting policy quality, where a metric with superior defect detection (AUROC 0.804) yields a significantly lower task success rate (13.3%) compared to a less effective metric (AUROC 0.638) that closely approximates the performance of an oracle-trained policy (90.0% vs. 93.3%). The authors highlight that many metrics rely on episode length as a misleading proxy for defect labeling, which can inflate AUROC scores and calls for a reevaluation of curation methods based on the policies they generate rather than merely the defects they identify.
The best defect-detection metric can lead to a policy that performs worse than a simpler alternative, challenging the assumptions of current curation practices.
We study whether demonstration-curation metrics that detect defective training episodes also improve the downstream behavior-cloning policy that trains on the curated data. On a contact-rich LIBERO pick-and-place benchmark with a controlled structural defect (early gripper release during the carry phase), we find that the two quantities are sharply decoupled. The metric with the highest defect-detection AUROC (0.804) produces the worst curated policy (13.3% task success), while a metric with a substantially lower AUROC (0.638) produces a policy that nearly matches the oracle trained on ground-truth clean data (90.0% vs. 93.3%). We further show that five of the seven metrics we evaluate exploit episode length as a trivial proxy for the defect label, a confound that inflates reported AUROCs to near-perfect values and disappears once episode length is controlled. Across all conditions, the contaminated baseline succeeds on only 3.3% of rollouts, and the two best curation methods close this to within 3 percentage points of the 93.3% oracle ceiling. Our results argue that curation methods should be evaluated by the policy they produce, not the defects they flag, and that any curation benchmark must control for episode length before reporting detection accuracy. We release the testbed, all metric implementations, and the evaluation pipeline.