Search papers, labs, and topics across Lattice.
The paper introduces "See & Switch," an interactive programming-by-demonstration framework that uses vision-based branching in conditional task graphs for robot skill learning. It employs eye-in-hand images to select between skill parts and detect out-of-distribution contexts, enabling efficient in-situ recovery demonstrations. Experiments on three manipulation tasks and user studies demonstrate the system's ability to perform branch selection and anomaly detection with high accuracy (90.7% and 87.9%, respectively).
Robots can now learn complex conditional skills from demonstrations by "seeing" the environment through eye-in-hand images and switching between learned behaviors, achieving high accuracy in branch selection and anomaly detection.
Programming robots by demonstration (PbD) is an intuitive concept, but scaling it to real-world variability remains a challenge for most current teaching frameworks. Conditional task graphs are very expressive and can be defined incrementally, which fits very well with the PbD idea. However, acting using conditional task graphs requires reliable perception-grounded online branch selection. In this paper, we present See & Switch, an interactive teaching-and-execution framework that represents tasks as user-extendable graphs of skill parts connected via decision states (DS), enabling conditional branching during replay. Unlike prior approaches that rely on manual branching or low-dimensional signals (e.g., proprioception), our vision-based Switcher uses eye-in-hand images (high-dimensional) to select among competing successor skill parts and to detect out-of-distribution contexts that require new demonstrations. We integrate kinesthetic teaching, joystick control, and hand gestures via an input-modality-abstraction layer and demonstrate that our proposed method is teaching modality-independent, enabling efficient in-situ recovery demonstrations. The system is validated in experiments on three challenging dexterous manipulation tasks. We evaluate our method under diverse conditions and furthermore conduct user studies with 8 participants. We show that the proposed method reliably performs branch selection and anomaly detection for novice users, achieving 90.7 % and 87.9 % accuracy, respectively, across 576 real-robot rollouts. We provide all code and data required to reproduce our experiments at http://imitrob.ciirc.cvut.cz/publications/seeandswitch.
