Search papers, labs, and topics across Lattice.
This paper introduces an immersive teleoperation framework that combines voice-controlled locomotion, VR-based manipulation, and bidirectional social interaction for controlling humanoid robots. By utilizing the Apple Vision Pro, operators can issue natural-language commands and receive egocentric visual feedback, resulting in novice users achieving 80% success in object manipulation and 70% in social interaction tasks after minimal training. The integration of LLM-assisted voice control and multimodal data collection supports future advancements in imitation learning and autonomy in humanoid robotics.
Novice users can achieve impressive success rates in robot manipulation and social tasks with minimal training, thanks to an immersive, language-assisted teleoperation system.
Humanoid robots can extend human presence to remote, constrained, or hazardous environments, but existing teleoperation interfaces often require physically demanding motion tracking or cognitively demanding low-level control. This paper presents an immersive teleoperation framework that integrates voice-controlled locomotion, VR-based manipulation, and bidirectional social interaction for whole-body humanoid control. Using Apple Vision Pro, the operator receives egocentric visual feedback, issues natural-language locomotion commands, and teleoperates the robot's arms and dexterous hands through wrist and finger tracking. An LLM-assisted voice-control module converts spoken instructions into high-level locomotion commands, while the manipulation module retargets human hand motions to the robot through inverse kinematics and PD control. The system also records multimodal data, including egocentric RGB observations, voice/text commands, joint states, hand motions, and eye-gaze signals, supporting future imitation learning and autonomy. We evaluate the framework on a Unitree H1 humanoid equipped with dexterous hands in manipulation and social interaction tasks. Results show that novice users can successfully operate the system after brief familiarization, achieving 80\% success in object manipulation and 70\% success in a social cube-passing task. These results demonstrate the potential of immersive, language-assisted teleoperation as an accessible interface for humanoid interaction, remote assistance, and multimodal data collection.
