Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study audits seven demonstration curation metrics to evaluate their effectiveness in identifying defects that degrade imitation-learning policies. Using a controlled testbed, the authors inject known defects into demonstrations and assess how well each metric separates defective from clean demonstrations and improves task success when used for curation. The findings reveal that while subtle perturbations can be detected by multivariate outlier scoring, structural errors are often missed by action-only metrics, leading to potentially misleading quality assessments that fail to enhance policy performance.
Action-only metrics can mislead curation efforts, scoring defective demonstrations as high quality and failing to improve imitation-learning policies.
Imitation-learning policies inherit the quality of the demonstrations they are trained on, and a growing set of curation metrics promise to score and filter low-quality demonstrations automatically. These metrics are each validated on different data with different protocols, so it is unclear which of them actually identify the demonstrations that harm a policy. We build a controlled testbed in which demonstration defects are injected with known type, and audit seven curation metrics along two axes: how well each separates defective from clean demonstrations, and whether training a behavior-cloning policy on each metric's curated subset improves task success. We study two defect regimes. Subtle perturbations (correlated action noise, tremor, truncation) are detectable by multivariate outlier scoring and, once removed, recover the full downstream gap. Structural errors, where the demonstration executes a wrong action at a key moment, are invisible to every action-only metric we test, and two of them are inverted: they score defective demonstrations as higher quality and, used for curation, tend to leave the policy at or below the uncurated baseline rather than above it. Only metrics that examine the state trajectory detect structural errors, and even the best of them recovers just a third of the downstream gap. High detection accuracy does not guarantee downstream improvement. We release the testbed and all curation implementations.