Search papers, labs, and topics across Lattice.
This study evaluates the temporal robustness of dexterous manipulation policies learned through imitation by comparing an expert and a learner across varying task execution speeds in the ParcelStow task. While both the scripted expert and the Action Chunking with Transformers (ACT) policy achieve 100% success at nominal speed, the ACT policy shows significant performance degradation at higher speeds, with success rates dropping from 100% to 53%, compared to a more modest decline from 100% to 84% for the expert. The findings highlight that achieving equal nominal success does not guarantee retention of expert-level performance under varying conditions, particularly in insertion tasks where the ACT policy struggles significantly.
Equal nominal success in dexterous manipulation doesn't mean your AI can handle speed variations like a human expert.
Dexterous manipulation policies learned by imitation are typically evaluated for robustness to variation in scenes, objects, or instructions, but their performance across task execution speeds is less often examined. This leaves open how much temporal robustness a learner retains relative to the expert it imitates. We compare an expert and learner under the same task conditions, initial-condition draws, and speedup factors. We instantiate the evaluation in ParcelStow, a contact-rich task in which the robot acquires, reorients, and inserts a parcel. The demonstrations span the speedup range for the manipulation phases after parcel acquisition. A scripted expert and an Action Chunking with Transformers (ACT) policy trained from the expert's demonstrations both achieve 100 percent task success at nominal speed. Their success rates diverge within the demonstrated range: at its maximum, expert success is 84 percent and ACT success is 53 percent. Two ACT policies with different parameter initializations show similar degradation, decreasing by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert. Stage-level analysis shows that 35 of ACT's 47 failures at the maximum demonstrated speed are insertion misalignments. Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, but only 64 percent complete the overall task, compared with 95 percent after expert acquisition. Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task. Equal nominal task success therefore does not imply preservation of expert performance across execution speeds. Code, data, and evaluation scripts are available at https://github.com/coenwerem/parcelstow.
