Search papers, labs, and topics across Lattice.
This paper introduces a multi-modal interaction framework for human-robot collaborative shelf-picking in warehouse environments, integrating gesture recognition, voice command interpretation, and visual/auditory feedback for enhanced teamwork. The framework leverages a Large Language Model (LLM) with Chain of Thought (CoT) reasoning and a physics-based simulation engine to enable safe retrieval of cluttered boxes and intelligent task planning. Real-world experiments validate the framework's effectiveness in gesture-guided box extraction, collaborative shelf clearing, and stability assistance.
LLMs and physics simulation enable robots to understand human gestures and voice commands for collaborative shelf-picking, making warehouse work safer and more efficient.
The growing presence of service robots in human-centric environments, such as warehouses, demands seamless and intuitive human-robot collaboration. In this paper, we propose a collaborative shelf-picking framework that combines multimodal interaction, physics-based reasoning, and task division for enhanced human-robot teamwork.The framework enables the robot to recognize human pointing gestures, interpret verbal cues and voice commands, and communicate through visual and auditory feedback. Moreover, it is powered by a Large Language Model (LLM) that utilizes Chain of Thought (CoT) and a physics-based simulation engine for safely retrieving cluttered stacks of boxes on shelves, a relationship graph for sub-task generation, extraction sequence planning, and decision making. Furthermore, we validate the framework through real-world shelf-picking experiments such as 1) Gesture-Guided Box Extraction, 2) Collaborative Shelf Clearing, and 3) Collaborative Stability Assistance. This work paves the way for more intuitive and effective human-robot collaboration in warehouse environments. A video demonstrating the real-world implementation of our proposed system is available at: https://youtu.be/353zmxMwESg?si=2_T-D7hUZnzl5522
