Search papers, labs, and topics across Lattice.
This paper introduces a human-robot collaboration (HRC) framework that uses Vision-Language Models (VLMs) to interpret human instructions and generate robot actions for assembly tasks. To mitigate VLM limitations like hallucination and failure anticipation, the framework incorporates a dual-correction mechanism: an internal model for pre-execution logical and task feasibility verification, and an external model for post-execution physical failure detection and rectification. Simulation and real-world experiments demonstrate that the proposed method improves the success rate of collaborative assembly tasks compared to baselines without correction models.
Robots can now better collaborate with humans on assembly tasks by using vision-language models augmented with internal and external correction mechanisms that improve task feasibility and handle physical failures.
Human-Robot Collaboration (HRC) plays an important role in assembly tasks by enabling robots to plan and adjust their motions based on interactive, real-time human instructions. However, such instructions are often linguistically ambiguous and underspecified, making it difficult to generate physically feasible and cooperative robot behaviors. To address this challenge, many studies have applied Vision-Language Models (VLMs) to interpret high-level instructions and generate corresponding actions. Nevertheless, VLM-based approaches still suffer from hallucinated reasoning and an inability to anticipate physical execution failures. To address these challenges, we propose an HRC framework that augments a VLM-based reasoning with a dual-correction mechanism: an internal correction model that verifies logical consistency and task feasibility prior to action execution, and an external correction model that detects and rectifies physical failures through post-execution feedback. Simulation ablation studies demonstrate that the proposed method improves the success rate compared to baselines without correction models. Our real-world experiments in collaborative assembly tasks supported by object fixation or tool preparation by an upper body humanoid robot further confirm the framewor's effectiveness in enabling interactive replanning across different collaborative tasks in response to human instructions, validating its practical feasibility.
