Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores a system for programming robots via natural language and gestures, leveraging LLMs and computer vision to translate human input into executable robot programs. The system incorporates a feedback loop with clarification questions and visual representations to enhance safety and transparency. The authors aim to improve user acceptance by enabling intuitive robot programming through natural human-robot interaction.
Imagine programming robots simply by talking and gesturing – this work brings that vision closer to reality.
This paper presents a novel concept for intuitive end-user programming of robots, inspired by natural interaction between humans. Natural language and supportive gestures are translated into robot programs using large language models (LLMs) and computer vision (CV). Through equally natural system feedback in the form of clarification questions and visual representations, the generated program can be reviewed and adjusted, thereby ensuring safety, transparency, and user acceptance.