Search papers, labs, and topics across Lattice.
This paper introduces a real-to-sim-to-real shared autonomy framework for improving fine-grained teleoperation by learning corrective behaviors. A k-nearest-neighbor (kNN) human surrogate, trained on minimal real-world data, models operator actions in simulation, enabling stable reinforcement learning of a residual copilot policy. Experiments with novice and experienced users on manipulation tasks demonstrate improved task success, efficiency, and higher-quality demonstrations for imitation learning compared to direct teleoperation and other shared autonomy approaches.
Just five minutes of real-world teleoperation data is enough to train a copilot that significantly boosts both novice and expert performance on complex manipulation tasks.
Fine-grained, contact-rich teleoperation remains slow, error-prone, and unreliable in real-world manipulation tasks, even for experienced operators. Shared autonomy offers a promising way to improve performance by combining human intent with automated assistance, but learning effective assistance in simulation requires a faithful model of human behavior, which is difficult to obtain in practice. We propose a real-to-sim-to-real shared autonomy framework that augments human teleoperation with learned corrective behaviors, using a simple yet effective k-nearest-neighbor (kNN) human surrogate to model operator actions in simulation. The surrogate is fit from less than five minutes of real-world teleoperation data and enables stable training of a residual copilot policy with model-free reinforcement learning. The resulting copilot is deployed to assist human operators in real-world fine-grained manipulation tasks. Through simulation experiments and a user study with sixteen participants on industry-relevant tasks, including nut threading, gear meshing, and peg insertion, we show that our system improves task success for novice operators and execution efficiency for experienced operators compared to direct teleoperation and shared-autonomy baselines that rely on expert priors or behavioral-cloning pilots. In addition, copilot-assisted teleoperation produces higher-quality demonstrations for downstream imitation learning.
