Search papers, labs, and topics across Lattice.
This paper introduces Static In, Dynamic Out (SIDO), a novel approach for adapting visuomotor policies to dynamic object manipulation by leveraging counterfactual action augmentation. By separating the manipulation task into predicting the future position of moving objects and adjusting actions accordingly, SIDO allows policies trained on static demonstrations to effectively handle unseen motion at test time. Experimental results across simulated and real-world tasks demonstrate that SIDO significantly enhances success rates in dynamic scenarios while maintaining performance on static tasks.
SIDO enables static-trained policies to excel in dynamic environments, achieving higher success rates in moving object manipulation without sacrificing static performance.
Visuomotor policies have advanced on manipulation tasks where the target object stays static during execution, but real deployments break this assumption: parts drift on conveyors and fruits sway in the wind. We introduce Static In, Dynamic Out (SIDO), a counterfactual action augmentation that enables a policy trained only on static object demonstrations to adapt to unseen object motion at test time. Our key idea is to factorize moving object manipulation into two sub-problems: predicting where the object will be, and reaching that predicted pose. SIDO displaces the object to a counterfactual future position and morphs the demonstrated action chunk to preserve the hand-object relative pose, yielding a goal-conditioned policy. At deployment an object pose predictor supplies the future position. Across three simulated tasks (Mug, Square, Stack) under five object motion patterns and two real-world tasks (Gantry, Peachtree), SIDO improves moving object success over the baselines while preserving static object performance. Project website: https://sido-staticindynamicout.github.io/.
