Search papers, labs, and topics across Lattice.
This paper presents a novel framework that utilizes Large Language Models (LLMs) to synthesize complex robotic actions from multimodal human inputs, including speech, gestures, and music. By integrating a speech transcription model, gesture recognition module, and beat detection pipeline, the system contextualizes these inputs to generate coherent action sequences for robotic execution. The results indicate that this approach significantly enhances the expressiveness and adaptability of human-robot interactions, moving towards more intuitive and context-aware robotic behaviors.
LLMs can transform how robots interpret and respond to human communication by seamlessly integrating speech, gestures, and music into action synthesis.
The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.
