Search papers, labs, and topics across Lattice.
This study introduces Fail-RAG, a Retrieval Augmented Generation framework designed for identifying failures in robotic operations within warehouse environments. By embedding failure images and contextual information to query against a failure database, Fail-RAG leverages Vision-Language Models to analyze and detail failures effectively. The framework demonstrated a significant improvement in failure detection accuracy, achieving an average increase of 25 percentage points over traditional VLMs across various robotic tasks.
Fail-RAG outperforms standard VLMs in detecting robotic failures, achieving a 25% boost in accuracy in real-world warehouse settings.
Industry automation is witnessing an evolution in robotics driven by both technological breakthroughs and societal changes: progress towards generalist robots, embodied and physical artificial intelligence (AI), and increasing labor shortage in manufacturing.An intelligent autonomous robot needs to not only act according to planned motions but also react to any unexpected events. In this study, we focus on such unexpected events in warehouses where robots are used for material handling. Specifically, we refer to any unexpected events as failures and develop methods to detect robot operations related failures. Rule-based detection methods may break since the form of failures could change due to the dynamic nature of both environments and tasks. We propose 'Fail-RAG', a Retrieval Augmented Generation (RAG)-based failure detection framework where failure images and context information are embedded and queried against a failure database by calculating their similarities. Vision-Language Models (VLMs) are further used to analyze failures and provide details by following our instruction template. We evaluated the performance of Fail-RAG by conducting both simulation and physical experiments using fixed robot arms and a mobile manipulator for multiple tasks that are common in warehouse automation. Fail-RAG achieved 25 percentage point higher failure detection accuracy on average across five types of robot operations compared to using off-the-shelf VLMs, indicating its effectiveness for real-world failure detection.
