Search papers, labs, and topics across Lattice.
TeleDex is introduced as an open-source system for dexterous robot teleoperation using only a standard phone, streaming 6-DoF wrist poses and 21-DoF hand states. This system retargets the phone's pose and hand state estimates to robot arms and multi-fingered hands, eliminating the need for external tracking infrastructure. Evaluations in simulation and real-world tasks demonstrate TeleDex's effectiveness as a scalable interface for collecting demonstrations to fine-tune robot manipulation policies.
Forget expensive motion capture suits – TeleDex lets you teleoperate dexterous robots with just your phone.
Despite increasing dataset scale and model capacity, robot manipulation policies still struggle to generalize beyond their training distributions. As a result, deploying state-of-the-art policies in new environments, tasks, or robot embodiments often requires collecting additional demonstrations. Enabling this in real-world deployment settings requires tools that allow users to collect demonstrations quickly, affordably, and with minimal setup. We present TeleDex, an open-source system for intuitive teleoperation of dexterous hands and robotic manipulators using any readily available phone. The system streams low-latency 6-DoF wrist poses and articulated 21-DoF hand state estimates from the phone, which are retargeted to robot arms and multi-fingered hands without requiring external tracking infrastructure. TeleDex supports both a handheld phone-only mode and an optional 3D-printable hand-mounted interface for finger-level teleoperation. By lowering the hardware and setup barriers to dexterous teleoperation, TeleDex enables users to quickly collect demonstrations during deployment to support policy fine-tuning. We evaluate the system across simulation and real-world manipulation tasks, demonstrating its effectiveness as a unified scalable interface for robot teleoperation. All software and hardware designs, along with demonstration videos, are open-source and available at orayyan.com/teledex.
