Search papers, labs, and topics across Lattice.
The paper introduces Self-supervised Temporal Ensemble Advantage Modeling (STEAM), a novel method that leverages expert demonstrations to effectively distinguish reliable progress from failures in robot learning. By training an ensemble of temporal-offset predictors on frame pairs from expert trajectories, STEAM generates a scalar advantage score that conservatively evaluates mixed-quality rollout data. The method significantly enhances policy success rates across various real-world tasks, achieving improvements of up to 59% when integrated with CFGRL.
STEAM redefines how robots learn from mixed-quality data, achieving up to 59% higher success rates in real-world tasks by effectively identifying reliable progress.
Real-world robot learning increasingly relies on heterogeneous data, but demonstrations and rollouts often mix useful progress with stalls, corrections, and suboptimal behavior. Effective policy learning therefore requires frame-level advantages that distinguish reliable local progress from failures and regressions. We propose Self-supervised Temporal Ensemble Advantage Modeling (STEAM), a label-free method that learns such advantages from expert demonstrations. STEAM trains an ensemble of temporal-offset predictors on frame pairs within expert trajectories, using the normalized temporal offset between two frames as a self-supervised signal. Each predictor maps a frame pair to a distribution over temporal offsets, which is converted into a scalar advantage. STEAM then takes the minimum advantage across the ensemble to score mixed-quality rollout data conservatively. Across real-world bimanual towel folding, chip checkout, cola restocking, and single-arm pick-and-place tasks, STEAM identifies stalls, failures, and recoveries. When combined with CFGRL, STEAM further improves policy success rate by 59%, 54.3%, 23% and 16.2% over baselines, respectively.
Tsinghua AI